test_save_as_mofile fails with gettext 0.22+

[mgorny]

Tested against 8adc7f9. When running the test suite with gettext 0.22 or newer installed on the system, I'm getting the following test failure:

$ tox -e py311
py311: pip-23.3.2-py3-none-any.whl already present in /home/mgorny/.local/share/virtualenv/wheel/3.11/embed/3/pip.json
py311: wheel-0.42.0-py3-none-any.whl already present in /home/mgorny/.local/share/virtualenv/wheel/3.11/embed/3/wheel.json
py311: setuptools-69.0.3-py3-none-any.whl already present in /home/mgorny/.local/share/virtualenv/wheel/3.11/embed/3/setuptools.json
py311: install_deps> python -I -m pip install coverage
.pkg: install_requires> python -I -m pip install 'setuptools>=40.8.0' wheel
.pkg: _optional_hooks> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: install_requires_for_build_wheel> python -I -m pip install wheel
.pkg: prepare_metadata_for_build_wheel> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py311: install_package> python -I -m pip install --force-reinstall --no-deps /tmp/polib/.tox/.tmp/package/1/polib-1.2.0.tar.gz
py311: commands[0]> coverage run tests/tests.py
.................................................................................F.....
======================================================================
FAIL: test_save_as_mofile (__main__.TestPoFile.test_save_as_mofile)
Test for the POFile.save_as_mofile() method.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/polib/tests/tests.py", line 654, in test_save_as_mofile
    self.assertEqual(s1, s2)
AssertionError: b"\xd[276 chars]0\x001\x01\x00\x00\x06\x01\x00\x00&\x00\x00\x0[789 chars]\x00" != b"\xd[276 chars]0\x00,\x01\x00\x00\x06\x01\x00\x00&\x00\x00\x0[807 chars]\x00"

----------------------------------------------------------------------
Ran 87 tests in 1.078s

FAILED (failures=1)
py311: exit 1 (1.50 seconds) /tmp/polib> coverage run tests/tests.py pid=938030
.pkg: _exit> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  py311: FAIL code 1 (18.01=setup[16.51]+cmd[1.50] seconds)
  evaluation failed :( (18.16 seconds)

They pass with gettext 0.21.1.

[mgorny]

Gentle ping.

[angelabad]

Hi @izimobil , we have the same bug on Debian package build:

• https://bugs.debian.org/1074675

Could you please take a look?

Thanks

[s3v-]

Hi,
test fail with tests/test_iso-8859-15.po (encoding: iso-8859-15) but not with tests/test_utf8.po (encoding: utf8).

According to the gettext 0.22 changelog:

Portability:

On systems with musl libc, the *gettext() functions in libc now work with MO files generated from PO files with an encoding other than UTF-8. To this effect, the msgfmt program now converts the messages to UTF-8 encoding before storing them in a MO file. You can prevent this byusing the msgfmt --no-convert option.
On systems with musl libc, the *gettext() functions in libc now work with MO files generated from PO files with ISO C 99 <inttypes.h> format string directive macros. To this effect, the msgfmt program pre-expands strings with such macros. You can prevent this by using the msgfmt --no-redundancy option.

this change in tests/tests.py make test pass again:

             os.close(fd)
             po = polib.pofile(reffile, autodetect_encoding=False, encoding=encoding)
             po.save_as_mofile(tmpfile1)
-            subprocess.call([msgfmt, '--no-hash', '--endianness=%s' % sys.byteorder, '-o', tmpfile2, reffile])
+            subprocess.call([msgfmt, '--no-hash', '--endianness=%s' % sys.byteorder, '--no-convert', '-o', tmpfile2, reffile])            
             try:
                 f = open(tmpfile1, 'rb')
                 s1 = f.read()

Tested in Debian + gettext 0.22.5-2 + glibc 2.40.
Strange thing: changelog mentions musl libc but documentation does not:

‘--no-convert’

Don’t convert the messages to UTF-8 encoding. By default, messages are converted to UTF-8 encoding before being stored in a MO file; this helps avoiding conversions at run time, since nowadays most locales use the UTF-8 encoding.

For this reason I'm not entirely sure about the change since I don't know if the error belongs to test or is caused by a gettext bug.
Hope it helps as a starting point.

Kind Regards

[izimobil]

Thank you @s3v- : I will fix the test very soon and publish a new release.