Ideas for extending the user base

[robertbachmann]

I'm creating this ticket as a follow-up to this discussion, so that we can collect ideas here.

EDIT: Converted the ticket to a Forum discussion

[robertbachmann]

Here are some of my ideas.

User forum

A mailing-list/forum could be useful. IMO both Google Groups or the new
GitHub discussion feature

Pandoc/Markdown support

People often don't write TeX directly but rather use a tool like pandoc to convert from
Markdown -> TeX.

Currently pandoc supports LaTeX and ConTeXt as output formats.
I'm working on a pandoc custom lua-writer pandoc-optex that adds OpTeX support.
The first version will be online in a couple of days.

Gap analysis

I'm currently in the process of looking at various documents I've created with LaTeX to see
which features could be ported to OpTeX (the format itself or more likely an .opm module)

Overleaf

Overleaf seems to be quite popular with some people. I'm currently experimenting how
to get the Overleaf Editor to play nicely with OpTeX

[robertbachmann]

The first version will be online in a couple of days.

Due to day job related issues I did not get much done. Hopefully will be able to get a release out within the next 14 days.

Currently I still have to:

1. Finish the user documentation
2. Fix some special cases regarding PDF outlines
3. Fix some special cases regarding URLs
4. Implement class-based styling for Span and Div elements (probably will do this after the first release)
5. Implement a .cmd/.bat script for Windows (probably will do this after the first release)

My plan is to release a v0.0.1 that can be tested by all interested OpTeX users (that have Linux/BSD/MacOSX), and then do (4) and (5). Once this is done and user feedback has been implemented, I'll make another release and also announce it on the pandoc mailing list.

[olsak]

Note that my student Jan Martínek created pandoc module which converts from pandoc documents to OpTeX. See https://github.com/spasa47/panoptexum. The OpTeX reader isn't implemented yet.

[robertbachmann]

Nice. I linked it in the other thread #85.

[olsak]

Ad: User forum

It is good idea. It cannot be Google forum because I have no Google account. I didn't accept the user agreement suggested by Google, so I cannot have such account (http://petr.olsak.net/zkusenost-gog.html).

It should be at GitHub. But I don't know if it is a limitation for another possible users.

The question and answers can be realized at tex.stackexchange. The tag "optex" is here but not many questions. The reason shoud be: A) OpTeX users knows everything they need (because using OpTeX is simple) or B) the set of OpTeX users is almost empty :).

Ad: pandoc convertors

This is great suggestion. I was thinking about it too, see my document OMLS (http://petr.olsak.net/ftp/olsak/optex/omls.pdf). IMHO, authors should use Markdown (or something similar) and convert their documents to LaTeX/ConTeXt/OpTeX. If they need to use it only as "to PDF convertor" (a blackbox) then it does not matter what TeX format is used. But if Markdown is used only as a tool to pre-processing a document by authors, then the conversion to a .tex file is done, then a typesetter can add more information about conversion to PDF in the .tex file. He/she can control the processing to PDF in more detail, using macros. So, .tex file (and additional macro files) is the main source of the document where the superset of information needed for all processing of the document is present (processing to various formats, not only PDF). The .tex file should be archived for latter use and it is a "center or heard of the document". What TeX format is used does matter in this second case.

Ad: gap analysis

I am ready to help OpTeX users if they need any additional feature. Typically, such a requirement can be solved by a few macro lines, so I add the result to the OpTeX tricks page (http://petr.olsak.net/optex/optex-tricks.html). I do this continuously, so I didn't think about a special gap analysis. Maybe, some requirements needs more than few lines of the code, then the package .opm should be created. For example, one student at our faculty will prepare an OpTeX tool for advanced PDF features (annotations, sounds, etc.). This is his bachelor's thesis topic.I hope that his result will be a .opm package.

Ad: Overleaf

Overleaf is LaTeX oriented. One can use OpTeX here but with more effort before starting it. It is described in OpTeX trick 0022 (the last one on the www page). If it is possible to add a module which makes the Overleaf text Editor more OpTeX friendly, then this will be nice. I don't know about such possibility.

[cao-]

Comparison questions like this TeXStackExchange may help get users interested. Or maybe not :)

Regarding features that could be implemented in OpTeX, one that I see nowhere mentioned in OpTeX (it is mentioned in tex-nutshell) is an analogous to the LaTeX microtype package. Of course there are the primitive luatex commands for this purpose, which I also successfully managed to use without much knowledge on what I was doing, but an easier interface would be nice.

Regarding packages, when I have time I could try and write some styles for the slides, more or less copying some of the beamer styles, if it may be useful. Let me know.

[gromadan]

I think another obstruction why not to use optex might be that arXiv does not support it. Actually, arXiv does not support neither pdfTEX nor LuaTEX?

Do you know, why? I mean, they support pdflatex and plain tex, but not pdftex. What is the technical issue when they run texlive anyway? I've been googling how context users do it since context is quite popular tex format I guess and I found this page. But this means that also in this case you have to go through postscript and then convert into pdf. That's kind of annoying especially because you cannot use the PDF features such as simply changing the size of the paper. You can draw and use colours I guess, but you cannot do it the way opmac/optex does it...

To be honest, I think that as for scientific articles, latex is probably the best choice anyway. The unified structure and macros are a great advantage here I think. My experience with the "professional" typographers in the journals (especially with Springer) is that you need to send them basically ready to use source code, otherwise they are going to spoil everything they touch.

Nevertheless, there are many other scientific texts (lecture notes, surveys, books...), which you may want to upload to arXiv although they are not scientific articles. Here, it would be convenient to be able to use some other format.

[olsak]

It seems that arXiv supposes technology from the 90s: PostScipt, DVI. It supposes that plain TeX is here only to using this old technology. It supposes that only LaTeX is continuously developed and is able to produce PDF directly.
I don't know how the submission process works if there are incompatible versions of packages at user side and at arXiv side, or incompatible versions of LaTeX kernel (the LaTeX kernel has been substantially changed recently).

[olsak]

This is similar to the problem "which came first: the hen or the egg". It doesn't pay to do big projects without more users and there will be no more users without such projects.
I have several ideas about projects, but I know the eggs problem mentioned above:

• conversion modules for pandoc from/to optex -- markdown, html, latex.
• .bib files scanner implemented by Lua (for example old constructions {\~a} should be converted to ã using Lua pre-processor).
• more fonts support.
• more hi-syntax support.
• colors using LuaTeX's attributes.
• support for speciailities of various languages.

[vlasakm]

My idea to extending the userbase is to keep the LaTeXisms away.

Usually when somebody learns about LaTeX, they need to:

• download/install a distribution, probably TeX Live, and maybe even that is enough -- a specialized editor (like TeXworks or Texmaker) is what most people suggest
• choose one of the available engines (LaTeX, pdfLaTeX, XeLaTeX, LuaLaTeX), which is of course confusing, since the obvious choices for the uninformed are LaTeX ("I want to use LaTeX") or pdfLaTeX ("I want to generate .pdf not .dvi!)
• figure out the workflow ("is it three times pdflatex and then bibtex or what"), this includes learning about all the external programs (makeindex, bibtex) and their Unicode limitations and maybe their more modern replacements
• learn how TeX works with fonts/input, and why is it so hard to just use the font and input they have
• learn what packages to use (which depends on the previous two points -- the workflow and engine used), even for such basic things like inserting an images, colors or even just changing the layout

The ecosystem is really complicated due to historical baggage. Most people don't care about that, and preferably even shouldn't care about that. While OpTeX solves a lot of the previous points, I think it can go even further.

Used to Linux, I would expect from all software to just tell me install package X and run command X on file Y containing for example the input Z. The software would reasonably support my system (such as the possibility to use system fonts, respect to the usual cache and configuration file directories).

I created mmtex to accomodate that. It strives to the whole package -- the full OpTeX format including the goodies is there, except the fonts, which for now have to come from the system. To be fully usable I expect, that following is needed:

• extend system support to Windows and the BSDs (only Linux is supported now)
• include a possibility to install TeX Live / CTAN packages (a package manager), mostly for fonts and some generic macros (like TikZ, nodetree and dirtree - these are just the ones I needed on top of OpTeX)
• be installable by the usual ways of Linux/Windows/BSD systems (e.g. apt install mmtex)

Anyways the areas where I think OpTeX has to improve in order to solve the usual (La)TeX pain points listed above:

• get rid of any preloaded fonts (see e.g. some small changes tex-nutshell#1 (comment) for my reasoning), essentially this is the only way users can (and do) get consused by the font/input encoding mess, and having preloaded/nonpreloaded (obsolete/modern) font loading means two code paths with one being mostly untested
• get rid of all mentions of OPmac (it inherently suffers from some of the pain points - more formats, more historical encoding), if OpTeX is supposed to be the successor, it should be self contained, not just link to OPmac resources, which are in some cases limited to Czech

Two examples of the second point are:

• link to http://petr.olsak.net/opmac-prospectus.pdf from the OpTeX home page (http://petr.olsak.net/optex/), there should be an OpTeX prospectus
• Handle color using attributes #57 (comment), which is still not solved AFAICT

[olsak]

link to http://petr.olsak.net/opmac-prospectus.pdf from the OpTeX home page (http://petr.olsak.net/optex/), there should be an OpTeX prospectus

OK. Maybe, it would be better to create such a prospectus by somebody else than me. The OPmac prospectus should be only an inspiration.

[olsak]

I've run install-tl (for unix) from TeXlive just now and I've found that there is a menu item "set installation scheme" and there are subitems (among others): "full", "basic" (plain and latex), "minimal" (plain only), "ConTeXt", ... It would be great if there is a subitem "optex". It means optex, plainTeX?, minim, tikz, OTF fonts. First question: is it possible from technical point of view? Second question (more serious): will be accepted such idea by TeXlive developers? Is it reasonable?

[vlasakm]

It could theoretically be as simple as adding two files, scheme-optex and collection-optex, although I imagine that there shouldn't in general be many schemes, which are more prominent in the installer, so adding only a collection may be more realistic.

Example of a scheme and collection definition files:

https://git.texlive.info/texlive/tree/Master/tlpkg/tlpsrc/scheme-basic.tlpsrc
https://git.texlive.info/texlive/tree/Master/tlpkg/tlpsrc/collection-luatex.tlpsrc

I vaguely remember that collections should be disjoint and together cover all packages, with each package being in one collection and that preferably the package and its dependencies should be grouped in a collection (so that the collections are largely self contained)?

So these constraints should be verified and taken into account when constructing the set of "members" (dependencies) of the collection.

If we are going to dig into this, I think that it would be a good idea to remedy the following:

• optex package is missing a dependency on luaotfload (see tlpkg/tlpobj/optex.tlpobj)
• optexcount is not in the same collection as optex and should preferably be included in the "new" collection

I think that including only OpenType fonts will be hard, since packages can include them in addition to the Type 1 format and if they are in a separate package I don't think there is a collection of these "OpenType" packages only.

Secondly, languages are also problematic. They are by themselves put into collections (e.g. collection-langczechslovak), and these can depend on other collection (e.g. langczechslovak depends on collection-latex). But it should be enough to just depend on all of the hyph-$LANG packages, e.g. hyphen-czech, etc. That should give the equivalent of the CTAN hyph-utf8 package, which nowadays contains all hyphenation patterns.

Last time I needed to install TeX Live for OpTeX only I did something the following:

cd /tmp

curl -sLO 'http://mirror.ctan.org/systems/texlive/tlnet/install-tl-unx.tar.gz'

tar -xf install-tl-unx.tar.gz

cat > texlive.profile << 'EOF'
# file texlive.profile, customize the below directories as you like
selected_scheme scheme-infraonly
TEXDIR /tmp/texlive
TEXMFCONFIG $TEXMFSYSCONFIG
TEXMFHOME $TEXMFLOCAL
TEXMFLOCAL /tmp/texlive/texmf-local
TEXMFSYSCONFIG /tmp/texlive/texmf-config
TEXMFSYSVAR /tmp/texlive/texmf-var
TEXMFVAR $TEXMFSYSVAR
instopt_adjustpath 0
instopt_adjustrepo 1
instopt_letter 0
instopt_portable 1
instopt_write18_restricted 1
tlpdbopt_autobackup 0
tlpdbopt_create_formats 1
tlpdbopt_generate_updmap 1
tlpdbopt_install_docfiles 0
tlpdbopt_install_srcfiles 0
tlpdbopt_post_code 1
EOF

install-tl-*/install-tl -profile texlive.profile

PATH="/tmp/texlive/bin/x86_64-linux/:$PATH"

# add more packages to the below command or run the command again with other packages to install
tlmgr install optex luaotfload tikz lm lm-math stix2-otf inconsolata hyphen-english hyphen-czech

fmtutil --no-strict --all

Command to update: tlmgr update --self --all. On rare occasions regenerating format files can fix things: fmtutil --all.

A simple self contained setup which is quiet small (105 MiB for the above), relatively easy to use and maintain. No problem in having to remember whether to use fmtutil fmtutil-sys or fmtutil-user. No backups - small enough to just delete and reinstall. No documentation and source files, although it is possible to enable that (e.g. have tlpdbopt_install_docfiles 1 and install texdoc).

[omasanori]

While I am aware that it is so hard, it would be marvelous if we could typeset documents written in Chinese, Japanese and Korean languages. There are luatexja for Japanese and luatexko for Korean, though the both do not support OpTeX but plain TeX and LaTeX for now. They, unsurprisingly, load luaotfload directly or indirectly so an adjustment for OpTeX's font selection mechanism is at least necessary.

[omasanori]

Let me explain the background a bit.

In Japanese text, we usually choose different font families for European (or non-Japanese) characters and Japanese characters.
The following manuscript using the OpTeX trick 0062 can be used as a workaround for now.

\useOpTeX
\fontfam[Pagella]

\initunifonts
\addto\_fontfeatures{fallback=ipaex;}\_reloading
\directlua{luaotfload.add_fallback ("ipaex", {"IPAexMincho:;"} )}
\rm

Hello, world.

こんにちは、世界。

あのイーハトーヴォのすきとおった風、夏でも底に冷たさをもつ青いそら、うつくしい森で飾られたモリーオ市、郊外のぎらぎらひかる草の波。

\bye

The problem here is that the third paragraph does not wrap lines. Japanese text has no spaces between words and no hyphenation, so a typeset program have to insert line breaks with a different algorithm like one implemented in LuaTeX-ja.

The following manuscript uses the same paragraphs but to be processed by plain LuaTeX with LuaTeX-ja.

\input luatexja.sty

Hello, world.

こんにちは、世界。

あのイーハトーヴォのすきとおった風、夏でも底に冷たさをもつ青いそら、うつくしい森で飾られたモリーオ市、郊外のぎらぎらひかる草の波。

\bye

[olsak]

Sorry, I am not able to understand technical details not aims of luatexja. Maybe, you can try to ask somebody from LuaTeX-ja project team. The main concept of OpTeX is: it has its own luatexbase.lua, we don't want to load luatexbase.lua from LaTeX project. OpTeX has is own key-value macros, we don't want to load kyval form LaTeX, etc. We dont' want to load more stuff which does \input luatexja.sty (it loads 90 % of LaTeX code).

[omasanori]

No problem, I am sorry for confusing you. I'd mention that there are languages requiring language-specific layout algorithms and thus typesetting them with OpTeX is currently not practical. As you mentioned, adding OpTeX support to LuaTeX-ja is the way to go, not vice versa.

[omasanori]

Adding OpTeX support to TeX4ht would be cool.

[olsak]

If I understand what TeX4ht does: it reads dvi output. And dvi output doesn't support OpenType fonts (this is impossible for technical reasons). We don't want return back to the TeX history and use 8bit fonts and dvi output in OpTeX.
TeX4ht is based on old technology. Best way is to create a new program with similar features like TeX4ht, but with the support of today's fonts.

[JHonaker]

I realize this is a huge amount of work, but I've always been very jealous that your books are only written in Czech. I enjoy using OpTeX so much more than LaTeX because of the shallower level of abstracting away TeX details. I think an English version of TeXbook Naruby would probably draw a lot of people toward OpTeX.

Do you have any plans at all for that?

[olsak]

The TeXbook naruby is another huge amount of work done in 2001. I don't plan to translate it manually but if there is an artificial intelligence capable to translate it then I am ready to offer this text. But I doubt it will be well because this is very technical and very specific text.