Experimenting with OpTeX on LuaHBTeX

[robertbachmann]

Yesterday I tried to use OpTeX with HarfBuzz.

\fontfam[lm]

% from: CTAN (current Google release does not work)
\font\notoemojifont=file:NotoColorEmoji.ttf:mode=harf; at 8pt
% from: https://github.com/mozilla/twemoji-colr/releases
\font\twemjifont=file:Twemoji.Mozilla.ttf:mode=harf; at 8pt

Noto: {\notoemojifont ✅😀👍\char"1F98B}

Twemoji: {\twemjifont ✅😀👍\char"1F98B}

% From FontAwesome website
\font\fa=file:FontAwesome6Free-Solid-900.otf:mode=harf; at 8pt
Fontawesome contextual ligatures:\nl
\code{\{\\Green\\fa music download person location\}}\nl
{\Green\fa music download person location inbox}

\bye

Results:

.ini file I used:

% ophbtex.ini

% luahbtex -ini ophbtex.ini
% ln -s /usr/local/bin/luahbtex /usr/local/bin/ophbtex
\let\dump=\relax

\input optex.ini

\_def\_banner {This is OpTeX (Olsak's Plain TeX) on LuaHBTeX, version <\_optexversion>}%

\_dump

\_endcode % ------------------------------

[Udi-Fogiel]

I use LuaHBTeX all the time, as I need to typeset documents in Hebrew. Without it the nikud is usually misplaced.

[vlasakm]

Essentially luahbtex is just luatex linked with harfbuzz library. The only behavior difference in the LuaTeX engine itself is the fact that this library can be loaded with require in Lua. From what I remember, luaotfload tries to require the harfbuzz library and if it succeeds, it makes available the mode=harf; for \font. That should be the only difference in using luahbtex vs luatex binary for OpTeX.

In https://github.com/vlasakm/mmtex I chose to not link with harfbuzz by default because many years ago the harfbuzz version required by LuaTeX was not available everywhere, but already for a long time I am compiling and distributing binaries linked against harfbuzz on all major Linux distributions, thus essentially I am using ophbtex.

I think making optex always use luahbtex and thus making harfbuzz available to those that want to try it, or in fact need it, like @Udi-Fogiel (#150). OpTeX aspires to support many languages, not just english, and making harfbuzz available by default could help even people who need something more complicated than "latin with accents", until it is implemented in (high quality) Lua code in ConTeXt (which by the way, if available is much better with TeX, as it can be much more flexible and more smart about discretionaries, etc.). Though I have to admit, I have not followed the situation with non-latin scripts and ConTeXt / Harfbuzz for a few years.

Btw, change to luahbtex should be the matter of changing fmtutil.cnf. In TeX Live it looks like this: https://git.texlive.info/texlive/tree/Master/texmf-dist/web2c/fmtutil.cnf?id=7b6535bc2bafbc8588742b9a4f0ac86e206cdd97#n121 - if it's changed to luahbtex instead of luatex, the optex binary will be harfbuzz-enabled. One can of course change it in their own installation (with the user configuration file), but a global change for everyone using TeX Live should be done in this file: https://git.texlive.info/texlive/tree/Master/tlpkg/tlpsrc/optex.tlpsrc?id=7b6535bc2bafbc8588742b9a4f0ac86e206cdd97#n5, which is what is propagated to that TeX Live file.

[olsak]

I suppose that there is no additional difference between LuaTeX and LuaHBTeX. I.e., all changes done by maintainers of LuaTeX are automatically copied to LuaHBTeX. If this is true then I am surprised why we have two different binaries, i.e. two versions of format files in TeX distributions etc. A single binary LuaHBTeX with name LuaTeX should be sufficient and life will be much more simpler. LuaHBTeX enables to load harfbuzz (but it does not it by default) and users can initialize it for individual fonts by mode=harf (but it is not default). So, why we have LuaTeX without HB inside it? Historical reasons? If such reasons is out of date, then we can ask to LuaTeX maintainers to merge LuaTeX to LuaHBTeX. If this will be done then we need to do nothing in OpTeX.

I have nothing against running OpTeX with LuaHBTeX by default (with name optex). If the LuaHBTeX will be still an independent existence in TeX distributions then we have to ask the TeX live maintainers (and MikTeX maybe too) to do relevant changes in their distributions. Maybe there should be one issue: we can try a time tests with \fontfam[lm] and run it with and without HB binary in order to measure if the loading of the harfbuzz library will take some significant time which is useless for most users. If it is true then we can try to disable harfbuzz loading in luaotfload by default and to enable it explicitly by a \harfbuzz macro (for example).

[vlasakm]

I suppose that there is no additional difference between LuaTeX and LuaHBTeX. I.e., all changes done by maintainers of LuaTeX are automatically copied to LuaHBTeX.

I just rechecked whether my old knowledge is still correct. The difference between luatex binary and luahbtex binary is, that compiling luahbtex has an additional define: -DLUATEX_HARFBUZZ_ENABLED, and is statitically linked with libluaharfbuzz.a (lua module written in C making harfbuzz usable in Lua), and libharfbuzz.a (harfbuzz itself).

-DLUATEX_HARFBUZZ_ENABLED results in the following:

1. Compiled with libharfbuzz ; using is printed in luahbtex --credits (source).
2. The names LuaHBTeX and luahbtex are used instead of LuaTeX and luatex respectively. (source). These are used in things like banner and version / credits messages.
3. A reference to symbol luaopen_luaharfbuzz is craeted in the binary, allowing require("luaharfbuzz") in Lua to find the "compiled in" library (source).
4. luaharfbuzz global variable is injected, which makes the luaharfbuzz library also available through that name (source).

That's it! They are otherwise compiled from exactly the same sources. They are essentially not two different projects, but the same project compiled either with the optional harfbuzz feature (-DLUATEX_HARFBUZZ_ENABLED) or not.

If this is true then I am surprised why we have two different binaries, i.e. two versions of format files in TeX distributions etc. A single binary LuaHBTeX with name LuaTeX should be sufficient and life will be much more simpler. LuaHBTeX enables to load harfbuzz (but it does not it by default) and users can initialize it for individual fonts by mode=harf (but it is not default).

That's correct.

So, why we have LuaTeX without HB inside it? Historical reasons? If such reasons is out of date, then we can ask to LuaTeX maintainers to merge LuaTeX to LuaHBTeX. If this will be done then we need to do nothing in OpTeX.

I don't know why exactly it is the way it is. Backwards compatibility may be a concern. IIUC most changes to LuaTeX since 1.0 version have been attempting to not change the default behavior, i.e. keeping compatibility. And as LuaTeX is now in feature freeze, I don't think anyone will be for having just luatex for both the harfbuzz/non-harfbuzz use cases.

I guess one of the advantages of the current way is, that it is possible to compile LuaTeX fully without harfbuzz. But most people don't care about that, since they use binary provided by TeX Live / MikTeX which solve any problems that might be associated.

I have nothing against running OpTeX with LuaHBTeX by default (with name optex). If the LuaHBTeX will be still an independent existence in TeX distributions then we have to ask the TeX live maintainers (and MikTeX maybe too) to do relevant changes in their distributions. Maybe there should be one issue: we can try a time tests with \fontfam[lm] and run it with and without HB binary in order to measure if the loading of the harfbuzz library will take some significant time which is useless for most users. If it is true then we can try to disable harfbuzz loading in luaotfload by default and to enable it explicitly by a \harfbuzz macro (for example).

The overhead possible in LuaTeX itself is the fact, that the luahbtex has statically linked harfbuzz, and is thus a bit bigger, and may take very slightly longer time to be loaded into memory, etc. Then unfortunately luahbtex sets the global variable luaharfbuzz to the loaded luaharfbuzz module, which involves loading the module. The modern way would be to not preload the library, but let it be lazily loaded on first require("luaharfbuzz"). Otherwise harfbuzz functionality in luahbtex has no cost until anyone uses it.

luaotfload is the thing using luaharfbuzz for us. It actually checks whether luaharfbuzz is available by doing require("luaharfbuzz") (source), which means that it actually has a higher cost without luahbtex (because with luatex it has to search on disk in vain, whereas with luahbtex it's already present in memory). If the load is succesfull, then two additional Lua files are loaded (source): luaotfload-harf-define.lua and luaotfload-harf-define.lua and luaotfload-harf-plug.lua which contain the hooks that make mode=harf possible.

I'll try to benchmark the additional overhead of

\fontfam[lm]

Hello world!

\bye

I think something like \harfbuzz which would make harfbuzz available only sometimes is possible, but it would be kind of hacky. I hope the overhead is low enough to allow us to always use luahbtex.

[vlasakm]

I installed a fresh TeX Live minimal installation similarly to my guide here: #41 (reply in thread), with the addition of installing luahbtex and modifying texmf-dist/web2c/fmtutil.cnf to include the line ophbtex luahbtex - optex.ini, and only then ran fmtutil --no-strict --all (note that this is not a proper way to add formats, as this files gets overwritten by TeX Live, user config files belong elsewhere).

I used a few benchmarks:

% require-luaotfload.tex
\directlua { require "luaotfload" }

\bye

% initunifonts.tex
\initunifonts

\bye

% lm-fonts.tex
\fontfam[lm]

\bye

% lm-fonts-hello.tex
\fontfam[lm]

Hello world!

\bye

% lm-fonts-lipsum.tex
\fontfam[lm]

\lipsum[150]

\bye

And compared them with proper warmup and averaging over multiple runs using the excellent hyperfine tool:

$ hyperfine --warmup 20 --runs 30 -L inputfile require-luaotfload.tex,initunifonts.tex,lm-fonts.tex,lm-fonts-hello.tex,lm-fonts-lipsum.tex 'luatex -fmt texmf-var/web2c/luatex/optex.fmt {inputfile}' 'luahbtex -fmt texmf-var/web2c/luahbtex/ophbtex.fmt {inputfile}'
Benchmark 1: luatex -fmt texmf-var/web2c/luatex/optex.fmt require-luaotfload.tex
  Time (mean ± σ):      33.1 ms ±   0.1 ms    [User: 26.3 ms, System: 6.9 ms]
  Range (min … max):    32.7 ms …  33.4 ms    30 runs

Benchmark 2: luahbtex -fmt texmf-var/web2c/luahbtex/ophbtex.fmt require-luaotfload.tex
  Time (mean ± σ):      34.0 ms ±   0.2 ms    [User: 26.9 ms, System: 7.2 ms]
  Range (min … max):    33.6 ms …  34.5 ms    30 runs

Benchmark 3: luatex -fmt texmf-var/web2c/luatex/optex.fmt initunifonts.tex
  Time (mean ± σ):     190.0 ms ±   1.3 ms    [User: 170.9 ms, System: 18.7 ms]
  Range (min … max):   187.8 ms … 192.4 ms    30 runs

Benchmark 4: luahbtex -fmt texmf-var/web2c/luahbtex/ophbtex.fmt initunifonts.tex
  Time (mean ± σ):     199.0 ms ±   6.1 ms    [User: 174.9 ms, System: 23.6 ms]
  Range (min … max):   191.3 ms … 215.7 ms    30 runs

Benchmark 5: luatex -fmt texmf-var/web2c/luatex/optex.fmt lm-fonts.tex
  Time (mean ± σ):     295.1 ms ±   7.8 ms    [User: 263.1 ms, System: 30.9 ms]
  Range (min … max):   280.2 ms … 308.4 ms    30 runs

Benchmark 6: luahbtex -fmt texmf-var/web2c/luahbtex/ophbtex.fmt lm-fonts.tex
  Time (mean ± σ):     292.3 ms ±   1.9 ms    [User: 262.0 ms, System: 29.6 ms]
  Range (min … max):   289.3 ms … 296.5 ms    30 runs

Benchmark 7: luatex -fmt texmf-var/web2c/luatex/optex.fmt lm-fonts-hello.tex
  Time (mean ± σ):     308.6 ms ±   3.2 ms    [User: 277.3 ms, System: 30.7 ms]
  Range (min … max):   299.5 ms … 313.7 ms    30 runs

Benchmark 8: luahbtex -fmt texmf-var/web2c/luahbtex/ophbtex.fmt lm-fonts-hello.tex
  Time (mean ± σ):     312.6 ms ±   3.1 ms    [User: 279.9 ms, System: 32.0 ms]
  Range (min … max):   303.5 ms … 317.2 ms    30 runs

Benchmark 9: luatex -fmt texmf-var/web2c/luatex/optex.fmt lm-fonts-lipsum.tex
  Time (mean ± σ):     585.3 ms ±   2.8 ms    [User: 549.6 ms, System: 34.5 ms]
  Range (min … max):   580.4 ms … 591.4 ms    30 runs

Benchmark 10: luahbtex -fmt texmf-var/web2c/luahbtex/ophbtex.fmt lm-fonts-lipsum.tex
  Time (mean ± σ):     583.8 ms ±   1.8 ms    [User: 548.6 ms, System: 34.2 ms]
  Range (min … max):   580.8 ms … 587.4 ms    30 runs

Usually luahbtex is consistenly worse by a few milliseconds, but different runs vary a bit (I just show one as example). I would say the difference is not much bigger than noise or the varience. For non-trivial documents the cost is negligible.

[Udi-Fogiel]

Though I have to admit, I have not followed the situation with non-latin scripts and ConTeXt / Harfbuzz for a few years.

Here is an example file comparing the two modes for Hebrew (the font is from here)

\initunifonts

\font\node={file:KeterYG-Medium:script=hebr;}
\font\harf={file:KeterYG-Medium:script=hebr;mode=harf;}
\def\text{%
וּֽבְקֻצְרְכֶם֙ אֶת־קְצִ֣יר אַרְצְכֶ֔ם לֹ֧א תְכַלֶּ֛ה פְּאַ֥ת שָׂדְךָ֖ לִקְצֹ֑ר וְלֶ֥קֶט קְצִֽירְךָ֖ לֹ֥א תְלַקֵּֽט׃
וְכַרְמְךָ֙ לֹ֣א תְעוֹלֵ֔ל וּפֶ֥רֶט כַּרְמְךָ֖ לֹ֣א תְלַקֵּ֑ט לֶֽעָנִ֤י וְלַגֵּר֙ תַּעֲזֹ֣ב אֹתָ֔ם אֲנִ֖י יְהֹוָ֥ה אֱלֹהֵיכֶֽם׃
וְלֹֽא־תִשָּׁבְע֥וּ בִשְׁמִ֖י לַשָּׁ֑קֶר וְחִלַּלְתָּ֛ אֶת־שֵׁ֥ם אֱלֹהֶ֖יךָ אֲנִ֥י יְהֹוָֽה׃
לֹֽא־תַעֲשֹׁ֥ק אֶת־רֵֽעֲךָ֖ וְלֹ֣א תִגְזֹ֑ל לֹֽא־תָלִ֞ין פְּעֻלַּ֥ת שָׂכִ֛יר אִתְּךָ֖ עַד־בֹּֽקֶר׃
לֹא־תְקַלֵּ֣ל חֵרֵ֔שׁ וְלִפְנֵ֣י עִוֵּ֔ר לֹ֥א תִתֵּ֖ן מִכְשֹׁ֑ל וְיָרֵ֥אתָ מֵּאֱלֹהֶ֖יךָ אֲנִ֥י יְהֹוָֽה׃
לֹא־תַעֲשׂ֥וּ עָ֙וֶל֙ בַּמִּשְׁפָּ֔ט לֹא־תִשָּׂ֣א פְנֵי־דָ֔ל וְלֹ֥א תֶהְדַּ֖ר פְּנֵ֣י גָד֑וֹל בְּצֶ֖דֶק תִּשְׁפֹּ֥ט עֲמִיתֶֽךָ׃
לֹא־תֵלֵ֤ךְ רָכִיל֙ בְּעַמֶּ֔יךָ לֹ֥א תַעֲמֹ֖ד עַל־דַּ֣ם רֵעֶ֑ךָ אֲנִ֖י יְהֹוָֽה׃
לֹֽא־תִשְׂנָ֥א אֶת־אָחִ֖יךָ בִּלְבָבֶ֑ךָ הוֹכֵ֤חַ תּוֹכִ֙יחַ֙ אֶת־עֲמִיתֶ֔ךָ וְלֹא־תִשָּׂ֥א עָלָ֖יו חֵֽטְא׃
לֹֽא־תִקֹּ֤ם וְלֹֽא־תִטֹּר֙ אֶת־בְּנֵ֣י עַמֶּ֔ךָ וְאָֽהַבְתָּ֥ לְרֵעֲךָ֖ כָּמ֑וֹךָ אֲנִ֖י יְהֹוָֽה׃
}
\textdirection=1\pardirection=1\node\text

\harf\text

\bye

The node mode does not really cope with situations where there are several nikud or te'amim (like accents) on one base letter.
node:
harf:
node:
harf:
node:
harf:

Note though, for Arabic the node mode is probably better than for Hebrew, as LuaTeX was developed with Arabic in mind, but I don't think it can cope with all possible shapers, section 6.1 of luaotfload's manual.

why we have LuaTeX without HB inside it? Historical reasons? If such reasons is out of date, then we can ask to LuaTeX maintainers to merge LuaTeX to LuaHBTeX. If this will be done then we need to do nothing in OpTeX.

This is probably because of different philosophies. The integration of HarfBuzz into LuaTeX wasn't done by the LuaTeX team, and if I recall correctly, Hans does not want it. This is the reason ConTeXt does not use it (and reading how LuaMetaTeX works this is not really surprising). The LaTeX team did want to use it as their default engine for the LuaLaTeX format, so it is maintained and shipped with TeXLive.

[olsak]

I am considering to ask Karl berry to set default engine for OpTeX luahbtex in TeXlive 2025 (instead luatex, which is current setting). I hope. that this will do no changes of actual behavior of OpTeX, but it enables to simply use option mode=harf which opens new features. I did some tests and I have found only one mysterious problem described here.

What is your opinion about this decision?

[vlasakm]

What is your opinion about this decision?

I think such change makes OpTeX strictly more powerful, and that it's a good change.

I did some tests and I have found only one mysterious problem described here.

Thanks, a great mystery indeed. Try \directlua{collectgarbage = function() end}. See https://tex.stackexchange.com/a/735470/125126 and latex3/luaotfload#297.

[Udi-Fogiel]

It was a really nice read, so thank you for the observation and the analysis! If there are still concerns, wouldn't it be possible to create two formats and create a flag in the executable that chooses the engine? But saving it from TeXLive would be nice.

[vlasakm]

wouldn't it be possible to create two formats and create a flag in the executable that chooses the engine?

I am not sure if I fully understand the idea, but if you mean we would create such a executable, then it's kind of complicated (support for windows, linux, tex live, miktex all at once somehow).

Change to luatex could implement this, but I am not sure if that's something we want, and can get other people behind.

There are very little changes in luatex and luahbtex. luaotfload for example doesn't do much differently, except for trying to handle mode=harf, in this case I think it was a bug that normal mode (node) behaved differently with luahbtex.

[olsak]

I prefer simplicity, i.e. only single format named optex, newly linked to luahbtex in TeXlive 2025. IMHO, there is no reason to return to the luatex (without hb).

But I ran into another problem. The Overleaf projects with OpTeX include now the latexmkrc file with single line:

$lualatex = 'TEXINPUTS=./optex//:$TEXINPUTS luatex -fmt=optex %O %S';

But we need to change this script to something like this:

if $TLVERSION <= 2024 then
   OPTEXENGINE=luatex
else
   OPTEXENGINE=luahbtex
fi
$lualatex = 'TEXINPUTS=./optex//:$TEXINPUTS $OPTEXENGINE -fmt=optex %O %S';

because the latexmkrc script is present in user's projects independently what TeXlive version is selected from the Overleaf menu.

The file above includes only pseudocode for test of TeXlive version. Knows anybody what exactly should be done here?

[olsak]

FYI: Karl Berry confirmed to me that texlive 2025 (starting from tlpretest) will configure OpTeX with LuaHBTeX engine.

I solved (with help of my son) my question above concerned to Overleaf's latexmkrc. Because this file is a part of Perl latexmk script, we have to use Perl syntax:

if (`pdftex -version` =~ /.*\(TeX Live (\d+).*\).*/) { $tl_version = $1; }
if ( $tl_version lt '2025' ) { $optex_engine = 'luatex' ; }
else { $optex_engine = 'luahbtex' ; } 
$lualatex = "OPENTYPEFONTS=./fonts// $optex_engine -fmt=optex %O %S";
$pdflatex = "OPENTYPEFONTS=./fonts// $optex_engine -fmt=optex %O %S";

This maketexrc script should be used in Overleaf if we want to proces OpTeX documents with various selecting of TeXlive versions in Overleaf's menu.

[vlasakm]

@olsak Sorry for not replying earlier, I didn't get chance to look that deeply into it, and don't have much experience with Overleaf and/or latexmk.

My very initial idea was also about using Perl to our advantage, but if I understand correctly, the problem is not only about new documents, but all the old documents that already hardcode:

$lualatex = 'TEXINPUTS=./optex//:$TEXINPUTS luatex -fmt=optex %O %S';

Btw, I briefly tried to just use:

$lualatex = 'optex %O %S';

And it seems to work. Maybe Overleaf or latexmk updated something, and this is now allowed?

In any case, for making all the old documents work with the hardcoded:

$lualatex = 'TEXINPUTS=./optex//:$TEXINPUTS luatex -fmt=optex %O %S';

I think what we can do here, is to use the tex file searching (kpathsea) to our advantage. In texmf.cnf we see that:

TEXFORMATS = $TEXMFDOTDIR;$TEXMF/web2c{/$engine,}

I.e. when luatex is running, it will search for the format file X (-fmt X) in three directories:

1. .
2. $TEXMF/web2c/luatex
3. $TEXMF/web2c/

The second one is interesting, as for luahbtex it will be $TEXMF/web2c/luahbtex. If we can have optex.fmt in both directories, than:

luatex -fmt optex
luahbtex -fmt optex

would both work.

One way to do this is, is to have two entries for optex in fmtutil.cnf:

optex luatex - optex.ini
optex luahbtex - optex.ini

This is probably not really intended and/or well supported. But it works, as fmtutil now generates two optex.fmt files in directories above, and their corresponding logs have:

# actual command line used during this run
# luahbtex -ini   -jobname=optex -progname=optex optex.ini </dev/null

# actual command line used during this run
# luatex -ini   -jobname=optex -progname=optex optex.ini </dev/null

Even e.g. fmtutil --byfmt optex seems to work fine and generates both formats correctly.

With luatex -recorder -fmt optex '\fontfam[lm] test \bye' I see:

INPUT /tmp/texlive/texmf-var/web2c/luatex/optex.fmt

And with luahbtex -recorder -fmt optex '\fontfam[lm] test \bye'

INPUT /tmp/texlive/texmf-var/web2c/luahbtex/optex.fmt

But a problem is how to ensure that optex links to luahbtex, as that is what determintes whether optex is luatex -fmt optex or luahbtex -fmt optex. That seems to be handled by texlinks script, and inspecting it and watching the verbose output (--verbose), it just interprets fmutil.cnf line by line, so last "optex entry" wins:

$ texlinks --verbose
ln -s luatex /tmp/texlive/bin/x86_64-linux/dviluatex
ln -s luatex /tmp/texlive/bin/x86_64-linux/optex
ln -s luahbtex /tmp/texlive/bin/x86_64-linux/optex

So, it seems that if we want:

1. optex to be luahbtex
2. luatex -fmt optex to still work for Overleaf users using the old template

We just need two fmtutil.cnf entries. As I mention above, formats defined by a package in TeX Live are determined by optex.tlpsrc file, so it just needs a second execute AddFormat line I guess.

But I think this is just a terrible hack at best. I think it would be much more preferable if we could either:

1. Make all Overleaf users upgrade and use $lualatex = 'optex %O %S';
2. Make Overleaf recognize optex as a format, which would mean we could get away with not needing any latexmkrc hacks in the first place.

[olsak]

Setting $lualatex = 'optex %O %S'; is good idea. The first use of OpTeX was at a time when the optex command did not exist in TeXLive, thus the latexmkrc has had some more complicated setting. I remained "imprisoned" at that time and did not realize that the situation had been different for a long time.

Most important are the Overleaf templates [1] and [2]. Users can clone them and start with their projects. I corrected their latexmkrc. The old projects at Overleaf are linked to old TeXlive versions, so no problems will occur unless users keep old maketexrc but select newer (2025+) TeXlive from menu. If TeXlive will support also the variant optex.fmt for luatex (I am not explicitly need it) then there will be no problems with old latexmkrc at Overlef at all.

Make Overleaf recognize optex as a format, which would mean we could get away with not needing any latexmkrc hacks in the first place.

This is probably impossible. Overleaf team supports only one format LaTeX. Their give LaTeX tutorials, LaTeX consultation, their build-in text editor has LaTeX syntax highlighting and word completions etc.