Test suite summary again w/ latest CTAN snapshot (July 2025)

[hpreusse]

A few days ago I tested again if test suite is usable w/ TL 2025. Unfortunately it is not. Here are the latest test results with a CTAN snapshot as of end of July and latexml snapshot as of 27th of August:

#   Failed test 't/babel/csquotes'
#   at /tmp/autopkgtest.6uUjCt/build.bUC/src/blib/lib/LaTeXML/Util/Test.pm line 120.
# Difference at line 26 for t/babel/csquotes
#       got : '<para'
#  expected : '<section'

#   Failed test 't/babel/french'
#   at /tmp/autopkgtest.6uUjCt/build.bUC/src/blib/lib/LaTeXML/Util/Test.pm line 120.
# Difference at line 25 for t/babel/french
#       got : '<para'
#  expected : '<title>Francais</title>'

#   Failed test 't/babel/page545'
#   at /tmp/autopkgtest.6uUjCt/build.bUC/src/blib/lib/LaTeXML/Util/Test.pm line 120.
# Difference at line 19 for t/babel/page545
#       got : 'xml:id="p1">'
#  expected : 'class="ltx_align_left"'
# Looks like you failed 3 tests of 7.
t/81_babel.t ..............

[dginev]

Right, this is not necessarily a surprise in light of #2044, #2429, #2434 still being open.

Are you sure we need a new issue, or is this one a duplicate to those observations?

[dginev]

@hpreusse when you tested latexml here, did you use the new precompiled kernel, that is now available since #2525 ? The important new step is make formats, added between make and make test.

[hpreusse]

Right, this is not necessarily a surprise in light of #2044, #2429, #2434 still being open.

Are you sure we need a new issue, or is this one a duplicate to those observations?

Feel free to close the other two issues, which I've opened; I guess they are obsolete now. I just wanted to give some kind of status update.

[hpreusse]

@hpreusse when you tested latexml here, did you use the new precompiled kernel, that is now available since #2525 ? The important new step is make formats, added between make and make test.

No, I did not. I did insert that command, but the results are not promising. Mostly just a space character as difference. If you need the whole log let me know.

Test Summary Report
-------------------
t/05_tokenize.t         (Wstat: 3072 (exited 12) Tests: 18 Failed: 12)
  Failed tests:  2, 4-5, 7-8, 11-13, 15-18
  Non-zero exit status: 12
t/10_expansion.t        (Wstat: 7680 (exited 30) Tests: 47 Failed: 30)
  Failed tests:  2-4, 8-10, 12-14, 17-19, 23, 25, 27-31
                34-43, 47
  Non-zero exit status: 30
t/12_grouping.t         (Wstat: 512 (exited 2) Tests: 3 Failed: 2)
  Failed tests:  2-3
  Non-zero exit status: 2
t/20_digestion.t        (Wstat: 2304 (exited 9) Tests: 11 Failed: 9)
  Failed tests:  2-7, 9-11
  Non-zero exit status: 9
t/22_fonts.t            (Wstat: 5632 (exited 22) Tests: 24 Failed: 22)
  Failed tests:  2-16, 18-24
  Non-zero exit status: 22
t/30_encoding.t         (Wstat: 6656 (exited 26) Tests: 27 Failed: 26)
  Failed tests:  2-27
  Non-zero exit status: 26
t/32_keyval.t           (Wstat: 2048 (exited 8) Tests: 9 Failed: 8)
  Failed tests:  2-9
  Non-zero exit status: 8
t/33_keyval_options.t   (Wstat: 2816 (exited 11) Tests: 12 Failed: 11)
  Failed tests:  2-12
  Non-zero exit status: 11
t/40_math.t             (Wstat: 3584 (exited 14) Tests: 15 Failed: 14)
  Failed tests:  2-15
  Non-zero exit status: 14
t/50_structure.t        (Wstat: 9472 (exited 37) Tests: 44 Failed: 37)
  Failed tests:  2-5, 7-8, 10-25, 27-30, 32-35, 37-38, 40-44
  Non-zero exit status: 37
t/53_alignment.t        (Wstat: 4608 (exited 18) Tests: 23 Failed: 18)
  Failed tests:  2-13, 15, 17-19, 21, 23
  Non-zero exit status: 18
t/55_theorem.t          (Wstat: 1280 (exited 5) Tests: 6 Failed: 5)
  Failed tests:  2-6
  Non-zero exit status: 5
t/56_ams.t              (Wstat: 1536 (exited 6) Tests: 8 Failed: 6)
  Failed tests:  2-6, 8
  Non-zero exit status: 6
t/65_graphics.t         (Wstat: 1792 (exited 7) Tests: 8 Failed: 7)
  Failed tests:  2-8
  Non-zero exit status: 7
t/70_parse.t            (Wstat: 6400 (exited 25) Tests: 29 Failed: 25)
  Failed tests:  2-10, 12-24, 26, 28-29
  Non-zero exit status: 25
t/80_complex.t          (Wstat: 3328 (exited 13) Tests: 17 Failed: 13)
  Failed tests:  2-13, 16
  Non-zero exit status: 13
t/81_babel.t            (Wstat: 1536 (exited 6) Tests: 7 Failed: 6)
  Failed tests:  2-7
  Non-zero exit status: 6
t/82_moderncv.t         (Wstat: 512 (exited 2) Tests: 3 Failed: 2)
  Failed tests:  2-3
  Non-zero exit status: 2
t/84_slides.t           (Wstat: 512 (exited 2) Tests: 3 Failed: 2)
  Failed tests:  2-3
  Non-zero exit status: 2
t/91_latexmlc_api.t     (Wstat: 256 (exited 1) Tests: 3 Failed: 1)
  Failed test:  3
  Non-zero exit status: 1
t/92_profiles.t         (Wstat: 1536 (exited 6) Tests: 9 Failed: 6)
  Failed tests:  3-7, 9
  Non-zero exit status: 6
t/93_formats.t          (Wstat: 3840 (exited 15) Tests: 29 Failed: 15)
  Failed tests:  3, 5, 7, 9, 11-13, 15, 17, 19, 21, 23, 25
                27, 29
  Non-zero exit status: 15
t/95_complex_config.t   (Wstat: 1024 (exited 4) Tests: 5 Failed: 4)
  Failed tests:  2-5
  Non-zero exit status: 4
t/96_fatal.t            (Wstat: 512 (exited 2) Tests: 5 Failed: 2)
  Failed tests:  3, 5
  Non-zero exit status: 2
Files=35, Tests=466, 565 wallclock secs ( 0.14 usr  0.06 sys + 475.85 cusr  6.01 csys = 482.06 CPU)
Result: FAIL
make: *** [Makefile:1901: test_dynamic] Error 255

[dginev]

As a general comment, we may want to add the "latest CTAN snapshot test" to our CI tests, while allowing it to fail, as some regressions are expected from the bleeding edge.

As reported, the issue isn't particularly helpful, as we still need to get your setup on our own machines to fully troubleshoot and develop a patch. So far I could tell that "something regressed in babel", but that's about it.