testing fails on unix where python is not available (but only python3)

[kloczek]

I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix>
• run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pypandoc-1.8-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pypandoc-1.8-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pypandoc-1.8
collected 0 items

========================================================================== no tests ran in 0.01s ===========================================================================

[JessicaTegner]

TOtally honest here, I don't know much about rpm packages.

How do you get the new release? Through pip right?
Do you clone the repo, or use the official release through pypi?

[kloczek]

That issue has nothing to do with rpm.
You can reproduce that using oprocedure which I've described.
Just plese run pytest.

[JessicaTegner]

Ohh. The reason for it, is because examples, documentation files and tests has been removed from the pypi release from version 1.8, because of some conflicting names, when installing

[kloczek]

I'm not using pypu sdist but tar atogenerated from git tag. https://github.com/NicklasTegner/pypandoc/archive/refs/tags/v1.8.tar.gz

[JessicaTegner]

It's because when building, from the pyproject.toml file.
In 1.8 we have removed the test and other files.

I would suggest download and extracting the tar.gz, then running the tests, and lastly creating the wheel

[kloczek]

I see insise tar ball tests.py.

[JessicaTegner]

yes but they aren't included when you build. When the whl gets produced they aren't included.

[kloczek]

You can check what is inside autogenerated from git tag tar ball https://github.com/NicklasTegner/pypandoc/tree/v1.8

[JessicaTegner]

I know, and in the tarball they are, but my guess is, that when you run the build command, they aren't included, just like when I run python setup.py sdist, because of a change for version 1.8.

My suggestion would be to run pytest before building.

[kloczek]

Just tested 1.9 and looks like new two units are failing

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pypandoc-1.9-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pypandoc-1.9-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra tests.py --deselect tests.py::TestPypandoc::test_pdf_conversion
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.14, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pypandoc-1.9
collected 40 items / 1 deselected / 39 selected

tests.py ..........................FF...........                                                                                                                     [100%]

================================================================================= FAILURES =================================================================================
_____________________________________________________________ TestPypandoc.test_conversion_with_mixed_filters ______________________________________________________________

self = <tests.TestPypandoc testMethod=test_conversion_with_mixed_filters>

    def test_conversion_with_mixed_filters(self):
        markdown_source = "-0-"

        lua = """\
        function Para(elem)
            return pandoc.Para(elem.content .. {{"{0}-"}})
        end
        """
        lua = textwrap.dedent(lua)

        python = """\
        #!/usr/bin/env python

        from pandocfilters import toJSONFilter, Para, Str

        def func(key, value, format, meta):
            if key == "Para":
                return Para(value + [Str("{0}-")])

        if __name__ == "__main__":
            toJSONFilter(func)

        """
        python = textwrap.dedent(python)

        with closed_tempfile(".lua", lua.format(1)) as temp1, closed_tempfile(".py", python.format(2)) as temp2:
            with closed_tempfile(".lua", lua.format(3)) as temp3, closed_tempfile(".py", python.format(4)) as temp4:
>               output = pypandoc.convert_text(
                    markdown_source, to="html", format="md", outputfile=None, filters=[temp1, temp2, temp3, temp4]
                ).strip()

tests.py:381:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pypandoc/__init__.py:93: in convert_text
    return _convert_input(source, format, 'string', to, extra_args=extra_args,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

source = b'-0-', format = 'markdown', input_type = 'string', to = 'html', extra_args = (), outputfile = None
filters = ['/tmp/tmpsu9lufkd.lua', '/tmp/tmpgux03sxh.py', '/tmp/tmp96de3ep5.lua', '/tmp/tmphc5bl3mo.py'], verify_format = True, sandbox = True, cworkdir = None

    def _convert_input(source, format, input_type, to, extra_args=(),
                       outputfile=None, filters=None, verify_format=True,
                       sandbox=True, cworkdir=None):

        _check_log_handler()
        _ensure_pandoc_path()

        if verify_format:
            format, to = _validate_formats(format, to, outputfile)
        else:
            format = normalize_format(format)
            to = normalize_format(to)

        string_input = input_type == 'string'
        if not string_input:
            if isinstance(source, str):
                input_file = [source]
            else:
                input_file = source
        else:
            input_file = []
        args = [__pandoc_path, '--from=' + format]

        args.append('--to=' + to)

        args += input_file

        if outputfile:
            args.append("--output=" + str(outputfile))

        if sandbox:
            if ensure_pandoc_minimal_version(2,15): # sandbox was introduced in pandoc 2.15, so only add if we are using 2.15 or above.
                args.append("--sandbox")

        args.extend(extra_args)

        # adds the proper filter syntax for each item in the filters list
        if filters is not None:
            if isinstance(filters, string_types):
                filters = filters.split()
            f = ['--lua-filter=' + x if x.endswith(".lua") else '--filter=' + x for x in filters]
            args.extend(f)

        # To get access to pandoc-citeproc when we use a included copy of pandoc,
        # we need to add the pypandoc/files dir to the PATH
        new_env = os.environ.copy()
        files_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "files")
        new_env["PATH"] = new_env.get("PATH", "") + os.pathsep + files_path
        creation_flag = 0x08000000 if sys.platform == "win32" else 0 # set creation flag to not open pandoc in new console on windows

        old_wd = os.getcwd()
        if cworkdir and old_wd != cworkdir:
            os.chdir(cworkdir)

        p = subprocess.Popen(
            args,
            stdin=subprocess.PIPE if string_input else None,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            env=new_env,
            creationflags=creation_flag)

        if cworkdir is not None:
            os.chdir(old_wd)

        # something else than 'None' indicates that the process already terminated
        if not (p.returncode is None):
            raise RuntimeError(
                'Pandoc died with exitcode "%s" before receiving input: %s' % (p.returncode,
                                                                               p.stderr.read())
            )

        if string_input:
            try:
                source = cast_bytes(source, encoding='utf-8')
            except (UnicodeDecodeError, UnicodeEncodeError):
                # assume that it is already a utf-8 encoded string
                pass
        try:
            stdout, stderr = p.communicate(source if string_input else None)
        except OSError:
            # this is happening only on Py2.6 when pandoc dies before reading all
            # the input. We treat that the same as when we exit with an error...
            raise RuntimeError('Pandoc died with exitcode "%s" during conversion.' % (p.returncode))

        try:
            stdout = stdout.decode('utf-8')
        except UnicodeDecodeError:
            # this shouldn't happen: pandoc more or less guarantees that the output is utf-8!
            raise RuntimeError('Pandoc output was not utf-8.')

        try:
            stderr = stderr.decode('utf-8')
        except UnicodeDecodeError:
            # this shouldn't happen: pandoc more or less guarantees that the output is utf-8!
            raise RuntimeError('Pandoc output was not utf-8.')

        # check that pandoc returned successfully
        if p.returncode != 0:
>           raise RuntimeError(
                'Pandoc died with exitcode "%s" during conversion: %s' % (p.returncode, stderr)
            )
E           RuntimeError: Pandoc died with exitcode "83" during conversion: Error running filter /tmp/tmpgux03sxh.py:
E           Could not find executable python

pypandoc/__init__.py:418: RuntimeError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
/home/tkloczko
_____________________________________________________________ TestPypandoc.test_conversion_with_python_filter ______________________________________________________________

self = <tests.TestPypandoc testMethod=test_conversion_with_python_filter>

    def test_conversion_with_python_filter(self):
        markdown_source = "**Here comes the content.**"
        python_source = '''\
        #!/usr/bin/env python

        """
        Pandoc filter to convert all regular text to uppercase.
        Code, link URLs, etc. are not affected.
        """

        from pandocfilters import toJSONFilter, Str

        def caps(key, value, format, meta):
            if key == 'Str':
                return Str(value.upper())

        if __name__ == "__main__":
            toJSONFilter(caps)
        '''
        python_source = textwrap.dedent(python_source)
        with closed_tempfile(".py", python_source) as tempfile:
>           output = pypandoc.convert_text(
                markdown_source, to='html', format='md', outputfile=None, filters=tempfile
            ).strip()

tests.py:332:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pypandoc/__init__.py:93: in convert_text
    return _convert_input(source, format, 'string', to, extra_args=extra_args,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

source = b'**Here comes the content.**', format = 'markdown', input_type = 'string', to = 'html', extra_args = (), outputfile = None, filters = ['/tmp/tmpk2dzvkz1.py']
verify_format = True, sandbox = True, cworkdir = None

    def _convert_input(source, format, input_type, to, extra_args=(),
                       outputfile=None, filters=None, verify_format=True,
                       sandbox=True, cworkdir=None):

        _check_log_handler()
        _ensure_pandoc_path()

        if verify_format:
            format, to = _validate_formats(format, to, outputfile)
        else:
            format = normalize_format(format)
            to = normalize_format(to)

        string_input = input_type == 'string'
        if not string_input:
            if isinstance(source, str):
                input_file = [source]
            else:
                input_file = source
        else:
            input_file = []
        args = [__pandoc_path, '--from=' + format]

        args.append('--to=' + to)

        args += input_file

        if outputfile:
            args.append("--output=" + str(outputfile))

        if sandbox:
            if ensure_pandoc_minimal_version(2,15): # sandbox was introduced in pandoc 2.15, so only add if we are using 2.15 or above.
                args.append("--sandbox")

        args.extend(extra_args)

        # adds the proper filter syntax for each item in the filters list
        if filters is not None:
            if isinstance(filters, string_types):
                filters = filters.split()
            f = ['--lua-filter=' + x if x.endswith(".lua") else '--filter=' + x for x in filters]
            args.extend(f)

        # To get access to pandoc-citeproc when we use a included copy of pandoc,
        # we need to add the pypandoc/files dir to the PATH
        new_env = os.environ.copy()
        files_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "files")
        new_env["PATH"] = new_env.get("PATH", "") + os.pathsep + files_path
        creation_flag = 0x08000000 if sys.platform == "win32" else 0 # set creation flag to not open pandoc in new console on windows

        old_wd = os.getcwd()
        if cworkdir and old_wd != cworkdir:
            os.chdir(cworkdir)

        p = subprocess.Popen(
            args,
            stdin=subprocess.PIPE if string_input else None,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            env=new_env,
            creationflags=creation_flag)

        if cworkdir is not None:
            os.chdir(old_wd)

        # something else than 'None' indicates that the process already terminated
        if not (p.returncode is None):
            raise RuntimeError(
                'Pandoc died with exitcode "%s" before receiving input: %s' % (p.returncode,
                                                                               p.stderr.read())
            )

        if string_input:
            try:
                source = cast_bytes(source, encoding='utf-8')
            except (UnicodeDecodeError, UnicodeEncodeError):
                # assume that it is already a utf-8 encoded string
                pass
        try:
            stdout, stderr = p.communicate(source if string_input else None)
        except OSError:
            # this is happening only on Py2.6 when pandoc dies before reading all
            # the input. We treat that the same as when we exit with an error...
            raise RuntimeError('Pandoc died with exitcode "%s" during conversion.' % (p.returncode))

        try:
            stdout = stdout.decode('utf-8')
        except UnicodeDecodeError:
            # this shouldn't happen: pandoc more or less guarantees that the output is utf-8!
            raise RuntimeError('Pandoc output was not utf-8.')

        try:
            stderr = stderr.decode('utf-8')
        except UnicodeDecodeError:
            # this shouldn't happen: pandoc more or less guarantees that the output is utf-8!
            raise RuntimeError('Pandoc output was not utf-8.')

        # check that pandoc returned successfully
        if p.returncode != 0:
>           raise RuntimeError(
                'Pandoc died with exitcode "%s" during conversion: %s' % (p.returncode, stderr)
            )
E           RuntimeError: Pandoc died with exitcode "83" during conversion: Error running filter /tmp/tmpk2dzvkz1.py:
E           Could not find executable python

pypandoc/__init__.py:418: RuntimeError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
/home/tkloczko
============================================================================= warnings summary =============================================================================
pypandoc/pandoc_download.py:62
  /home/tkloczko/rpmbuild/BUILD/pypandoc-1.9/pypandoc/pandoc_download.py:62: DeprecationWarning: invalid escape sequence \.
    regex = re.compile(r"/jgm/pandoc/releases/download/.*(?:"+processor_architecture+"|x86|mac).*\.(?:msi|deb|pkg)")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================= short test summary info ==========================================================================
FAILED tests.py::TestPypandoc::test_conversion_with_mixed_filters - RuntimeError: Pandoc died with exitcode "83" during conversion: Error running filter /tmp/tmpgux03sxh...
FAILED tests.py::TestPypandoc::test_conversion_with_python_filter - RuntimeError: Pandoc died with exitcode "83" during conversion: Error running filter /tmp/tmpk2dzvkz1...
========================================================== 2 failed, 37 passed, 1 deselected, 1 warning in 4.90s ===========================================================

[kloczek]

I know, and in the tarball they are, but my guess is, that when you run the build command, they aren't included, just like when I run python setup.py sdist, because of a change for version 1.8.

My suggestion would be to run pytest before building.

On typical rpm package build test suite is always executed after builds and install.

[JessicaTegner]

So for your errors, it seems that in both instances, it can't find the "python" executable when trying to use a python filter.
What do you say would be the best solutions? Trying "python3" before "python", since we actually want py3, or the other way around, where we try python3, if regular "python" executable couldn't be found?

[kloczek]

So for your errors, it seems that in both instances, it can't find the "python" executable when trying to use a python filter.
What do you say would be the best solutions?

Instead hardcoding "python" executable name use sys.executable.

[JessicaTegner]

Instead hardcoding "python" executable name use sys.executable.

We are not hardcoding the name per say. THe error is from the shibang lines when we test with a python filter.