Allow to use more recent kernel on interactive_tool_jupyter_notebook

[yvanlebras]

Describe the bug
Not really a bug BUT using interactive_tool_jupyter_notebook tool, we see that R kernel (and others I guess) are using old version:

You are using R version:    	 4.0.3 (2020-10-10)
And the latest R version is:	 4.4.1 (2024-06-15)

It is thus difficult or impossible to install recent packages

Galaxy Version and/or server at which you observed the bug
usegalaxy.eu

Expected behavior
it seems to me important to update regularly this tool and thus related docker image so 1/ we can use recent packages/libraries 2/ we allow, using previous versions of the tool, to still use jupitool on worklows using nobooks as input datasets

Additional context
need to take care to keep possibilities to use old versions for jupytool use in workflows

[yvanlebras]

Timothée
TIMOTHEE

(first try of computer writing of my little one ;) )

[bgruening]

The Docker image would need an update: https://github.com/bgruening/docker-jupyter-notebook

I guess bumping all requirements or just triggering a new build should do the trick.

[PaulineSGN]

Hello @yvanlebras and @bgruening !

It seems that the Docker image was updated with the new version of the tool (1.0.1) — thank you, @bgruening ! However, I've encountered an issue with this new version. It appears the error is related to the kernel:

[TrustNotebookApp] Writing notebook-signing key to /home/jovyan/.local/share/jupyter/notebook_secret
[NbConvertApp] Converting notebook pivot_wider_jupytool_notebook.ipynb to notebook
[NbConvertApp] WARNING | Kernelspec name ir cannot be found!
[NbConvertApp] ERROR | No such kernel named ir

This error occurs only when the "Execute notebook and return a new one." option is set to "Yes."

Let me know if you have any ideas on how to fix this problem.
Thanks!

[yvanlebras]

up @bgruening any idea why there is this error on "ir" kernel now you updated the image? Seems to me maybe this is "just" a typo as for me, ir is maybe not the good name of kernel, maybe it is r ?

[bgruening]

Where do you see this?

[yvanlebras]

Thank you Björn for coming back to us! Here is the stderr of this history https://ecology.usegalaxy.eu/u/ylebras/h/tuto-ecoregionalization step 59:

          [TrustNotebookApp] Writing notebook-signing key to /home/jovyan/.local/share/jupyter/notebook_secret
[NbConvertApp] Converting notebook pivot_wider_jupytool_notebook.ipynb to notebook
[NbConvertApp] WARNING | Kernelspec name ir cannot be found!
[NbConvertApp] ERROR | No such kernel named ir
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/jupyter_client/manager.py", line 87, in wrapper
    out = await method(self, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_client/manager.py", line 435, in _async_start_kernel
    kernel_cmd, kw = await self._async_pre_start_kernel(**kw)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_client/manager.py", line 397, in _async_pre_start_kernel
    self.kernel_spec,
    ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_client/manager.py", line 195, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_client/kernelspec.py", line 285, in get_kernel_spec
    raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named ir
Traceback (most recent call last):
  File "/opt/conda/bin/jupyter-nbconvert", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_core/application.py", line 283, in launch_instance
    super().launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.11/site-packages/traitlets/config/application.py", line 1075, in launch_instance
    app.start()
  File "/opt/conda/lib/python3.11/site-packages/nbconvert/nbconvertapp.py", line 420, in start
    self.convert_notebooks()
  File "/opt/conda/lib/python3.11/site-packages/nbconvert/nbconvertapp.py", line 597, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/opt/conda/lib/python3.11/site-packages/nbconvert/nbconvertapp.py", line 563, in convert_single_notebook
    output, resources = self.export_single_notebook(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nbconvert/nbconvertapp.py", line 487, in export_single_notebook
    output, resources = self.exporter.from_filename(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nbconvert/exporters/exporter.py", line 201, in from_filename
    return self.from_file(f, resources=resources, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nbconvert/exporters/exporter.py", line 220, in from_file
    return self.from_notebook_node(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nbconvert/exporters/notebook.py", line 36, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nbconvert/exporters/exporter.py", line 154, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nbconvert/exporters/exporter.py", line 353, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nbconvert/preprocessors/base.py", line 48, in __call__
    return self.preprocess(nb, resources)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nbconvert/preprocessors/execute.py", line 97, in preprocess
    with self.setup_kernel():
  File "/opt/conda/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nbclient/client.py", line 600, in setup_kernel
    self.start_new_kernel(**kwargs)
  File "/opt/conda/lib/python3.11/site-packages/jupyter_core/utils/__init__.py", line 165, in wrapped
    return loop.run_until_complete(inner)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nbclient/client.py", line 550, in async_start_new_kernel
    await ensure_async(self.km.start_kernel(extra_arguments=self.extra_arguments, **kwargs))
  File "/opt/conda/lib/python3.11/site-packages/jupyter_core/utils/__init__.py", line 198, in ensure_async
    result = await obj
             ^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_client/manager.py", line 96, in wrapper
    raise e
  File "/opt/conda/lib/python3.11/site-packages/jupyter_client/manager.py", line 87, in wrapper
    out = await method(self, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_client/manager.py", line 435, in _async_start_kernel
    kernel_cmd, kw = await self._async_pre_start_kernel(**kw)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_client/manager.py", line 397, in _async_pre_start_kernel
    self.kernel_spec,
    ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_client/manager.py", line 195, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_client/kernelspec.py", line 285, in get_kernel_spec
    raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named ir

And @PaulineSGN sent an error from inside eu also if I am not wrong...

[bgruening]

Is that on old notebook created with another Jupyter Notebook instance?

Or was this notebook that you try to load generated with the same version?

[yvanlebras]

oh, you're right... I think it is an old notebook... Will try to update the notebook and come back to you if needed!

[yvanlebras]

Seems to me this issue is now resolved as now we can easily reload the tool version we were using. Now, this Galaxy bug is solved (don't find the related issue created by @mvdbeek ) we can go forward on other issues related to 1/ the name of the kernel who are changing between jupyter notebook images 2/ conserve all the kernel as "real jupyter kernel" as it seems for exemple in the last version of jupyter notebook tool https://github.com/usegalaxy-eu/galaxy/blob/release_24.1_europe/tools/interactive/interactivetool_jupyter_notebook_1.0.1.xml#L3 there is no more R kernel so we can't use this tool in "batch mode" with R code => see issue #266