Failing Deploy to GitHub Pages due to ModuleNotFoundError

[chiaraf10]

Hello,

In my repo, I am using Pygem library ( https://mathlab.github.io/PyGeM/ ). Of this library there is no release in pip or conda. I am able to run it locally since I have followed the installation by running python setup.py install. I have cloned the repo and I am storing it inside the nbs folder so that I can call the script from the notebooks.
My problem begins when I am pushing my notebooks to Github and the Deploy pages action fails due to error: "ModuleNotFoundError: No module named 'pygem'".

any help on how I would fix this?

Here is the traceback:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3433, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-68dd88ca3b53>", line 1, in <module>
    from pygem.rbf import RBF
ModuleNotFoundError: No module named 'pygem'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/process.py", line 205, in _process_chunk
    return [fn(*args) for args in chunk]
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/process.py", line 205, in <listcomp>
    return [fn(*args) for args in chunk]
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/fastcore/parallel.py", line 46, in _call
    return g(item)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/serve_drv.py", line 22, in main
    if src.suffix=='.ipynb': exec_nb(src, dst, x)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/serve_drv.py", line 16, in exec_nb
    cb()(nb)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/processors.py", line 231, in __call__
    def __call__(self, nb): return self.nb_proc(nb).process()
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/process.py", line 126, in process
    for proc in self.procs: self._proc(proc)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/process.py", line 119, in _proc
    for cell in self.nb.cells: self._process_cell(proc, cell)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/process.py", line 110, in _process_cell
    if callable(proc) and not _is_direc(proc): cell = opt_set(cell, proc(cell))
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/processors.py", line 201, in __call__
    raise Exception(f"Error{' in notebook: '+title if title else ''} in cell {cell.idx_} :\n{cell.source}") from self.k.exc[1]
Exception: Error in notebook: Morphing. in cell 6 :
from pygem.rbf import RBF
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.8/x64/bin/nbdev_docs", line 8, in <module>
    sys.exit(nbdev_docs())
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/fastcore/script.py", line 119, in _f
    return tfunc(**merge(args, args_from_prog(func, xtra)))
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/quarto.py", line 226, in nbdev_docs
    cache,cfg,path = _pre_docs(path, n_workers=n_workers, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/quarto.py", line 174, in _pre_docs
    cache = proc_nbs(path, n_workers=n_workers, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/serve.py", line 76, in proc_nbs
    parallel(nbdev.serve_drv.main, files, n_workers=n_workers, pause=0.01, **kw)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/fastcore/parallel.py", line 117, in parallel
    return L(r)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/fastcore/foundation.py", line 98, in __call__
    return super().__call__(x, *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/fastcore/foundation.py", line 106, in __init__
    items = listify(items, *rest, use_list=use_list, match=match)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/fastcore/basics.py", line 66, in listify
    elif is_iter(o): res = list(o)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/process.py", line 570, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/_base.py", line [403](https://github.com/chiaraf10/SSMT/actions/runs/3370141521/jobs/5590669074#step:2:414), in __get_result
    raise self._exception
Exception: Error in notebook: Morphing. in cell 6 :
from pygem.rbf import RBF
Error: Process completed with exit code 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/process.py", line 205, in _process_chunk
    return [fn(*args) for args in chunk]
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/process.py", line 205, in <listcomp>
    return [fn(*args) for args in chunk]
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/fastcore/parallel.py", line 46, in _call
    return g(item)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/serve_drv.py", line 22, in main
    if src.suffix=='.ipynb': exec_nb(src, dst, x)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/serve_drv.py", line 16, in exec_nb
    cb()(nb)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/processors.py", line 231, in __call__
    def __call__(self, nb): return self.nb_proc(nb).process()
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/process.py", line 126, in process
    for proc in self.procs: self._proc(proc)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/process.py", line 119, in _proc
    for cell in self.nb.cells: self._process_cell(proc, cell)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/process.py", line 110, in _process_cell
    if callable(proc) and not _is_direc(proc): cell = opt_set(cell, proc(cell))
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/processors.py", line 201, in __call__
    raise Exception(f"Error{' in notebook: '+title if title else ''} in cell {cell.idx_} :\n{cell.source}") from self.k.exc[1]
Exception: Error in notebook: Morphing. in cell 6 :
from pygem.rbf import RBF
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.8/x64/bin/nbdev_docs", line 8, in <module>
    sys.exit(nbdev_docs())
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/fastcore/script.py", line 119, in _f
    return tfunc(**merge(args, args_from_prog(func, xtra)))
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/quarto.py", line 226, in nbdev_docs
    cache,cfg,path = _pre_docs(path, n_workers=n_workers, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/quarto.py", line 174, in _pre_docs
    cache = proc_nbs(path, n_workers=n_workers, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/nbdev/serve.py", line 76, in proc_nbs
    parallel(nbdev.serve_drv.main, files, n_workers=n_workers, pause=0.01, **kw)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/fastcore/parallel.py", line 117, in parallel
    return L(r)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/fastcore/foundation.py", line 98, in __call__
    return super().__call__(x, *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/fastcore/foundation.py", line 106, in __init__
    items = listify(items, *rest, use_list=use_list, match=match)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/fastcore/basics.py", line 66, in listify
    elif is_iter(o): res = list(o)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/process.py", line 570, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/concurrent/futures/_base.py", line [403](https://github.com/chiaraf10/SSMT/actions/runs/3370141521/jobs/5590669074#step:2:414), in __get_result
    raise self._exception
Exception: Error in notebook: Morphing. in cell 6 :
from pygem.rbf import RBF
Error: Process completed with exit code 1.

PS. I cannot share the repo, as I am storing some private data.

[deven-gqc]

@chiaraf10 Did you add the pygem to requirements in settings.ini ?

[chiaraf10]

yes. I tried and it doesn´t work.
Pygem doesn´t have pip or conda installation therefore if I run with pygem as requirement in setting.ini I get the error:

ERROR: Could not find a version that satisfies the requirement pygem (from ssmt[dev]) (from versions: none)
ERROR: No matching distribution found for pygem
Error: Process completed with exit code 1.

I tried to run the setup.py in one cell before the import as !python install setup.py but the deploy page still fails.
Alternatevely I also tried not to install the dependency but only call the function I need, as it is a single .py file.

The pygem folder is located inside the nbs folder (with init.py inside). and it is import as from pygem.file import function.
It works locally but fails the test remotely.

[deven-gqc]

If you add this command to your deploy.yml file, I think it should stop giving you the error.

pip install git+https://github.com/mathLab/PyGeM

[muellerzr]

@hamelsmu @seeM I think this is the same issue I'm finding with nbdev-extensions when I deploy and the docs should be built with extensions from the library itself (see here: https://github.com/muellerzr/nbdev-extensions/actions/runs/3491673241/jobs/5844583540), however my deploy.yaml already installs via pip install -e .

[hamelsmu]

I'm looking at this now

[hamelsmu]

Note: @muellerzr's issue was resolved and was unrelated to this issue.

[chiaraf10]

Thank you for all the suggestions.
I have managed to find a workaround.
Pygem has MIT licence therefore I was able to export part of the code and create a notebook in the nbdev format (by citing the source).
Then I exported it as py. module with the command nbdev_prepare. The Github Deploy pages seems to work fine.

Before, I did try to modify the deploy.yml file as suggested:

from:
                name: Deploy to GitHub Pages
                on:
                  push:
                    branches: ["main", "master"]
                  workflow_dispatch:
                jobs:
                  deploy:
                    runs-on: ubuntu-latest
                    steps:
                      - name: Quarto
                        uses: fastai/workflows/quarto-ghp@master

to

              name: Deploy to GitHub Pages
              on:
                push:
                  branches: ["main", "master"]
                workflow_dispatch:
              jobs:
                deploy:
                  runs-on: ubuntu-latest
                  steps:
                    - name: Upgrade pip
                      run: python -m pip install --upgrade pip
              
                    - name: Install Pygem
                      run: pip install git+https://github.com/mathLab/PyGeM
              
                    - name: Quarto
                      uses: fastai/workflows/quarto-ghp@master

And when pushing to the main, it was installing the library, but the error was still there (see the image)
I supposed it had to be related on how it is imported. But I couldn´t find the exact solution to this.

Also, I tried to import the functions I needed as .py module only (so I wouldn´t need to install the library). And I was getting the same traceback ..

[batmanscode]

Having a similar issue. The package is in requirements and installed via pypi but I still get a module not found error. Link to action: https://github.com/batmanscode/LinkNinja/actions/runs/3720662974/jobs/6310286199

[ncoop57]

Is there any easy way to just add a git repo to be installed in the settings.ini requirements var?

Naively adding the git+https://... to requirements var produces the following error:

      error in doremi setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Parse error at "'+https:/'": Expected string_end

[Taytay]

@ncoop57, I just ran across this same error, and found that this is the correct syntax to use when specifying "git+https" as a requirement:

requirements =
    "PyGem @ git+https://github.com/mathLab/PyGeM"

Apparently the "name" of the package (the part before the @) is arbitrary, but setuptools recognizes this syntax at install timel.

I used it to pin my version of Pandera to a particular commit:
"pandera @ git+https://github.com/unionai-oss/pandera@c240b1bf015ffbb41c6b6c94471729bc507ce9e9"