Security/#346 dependency upgrade (#347)

* #346: Dependency upgrade
* Updated notebook-dependencies
* Added documentation about various dependencies of the AI-Lab
* Fixed dependencies around ITCD, SLCT
  affecting notebook script_languages_container and the related notebook-test.
  Please see file test/notebooks/test_dependencies.txt for details.
* Set release date
* [CodeBuild]

Author: ckunki

.github/actions/prepare_test_env/action.yml

@@ -21,8 +21,12 @@ runs:
      shell: bash
 
    - name: Show available disk space
+     shell: bash
      run: df -h
+
+   - name: Allow unprivileged user namespaces
      shell: bash
+     run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
 
    - name: Setup Python & Poetry Environment
      uses: exasol/python-toolbox/.github/actions/python-environment@0.14.0

doc/changes/changes_3.2.0.md

@@ -1,15 +1,40 @@
-# AI-Lab 3.2.0 released ????-??-??
+# AI-Lab 3.2.0 released 2025-01-16
 
-Code name:
+Code name: Additional Updates on top of 3.1.0
 
 ## Summary
 
+This release updates dependencies and fixes security vulnerabilities on top of 3.1.0.
+
+Fixed vulnerabilities:
+
+* Vulnerabilities in direct dependency `jinja2` version 3.1.4
+  * #50 Moderate: Jinja has a sandbox breakout through malicious filenames Moderate
+  * #49 Moderate: Jinja has a sandbox breakout through indirect reference to format method Moderate
+* Vulnerabilities in transitive dependency `ansible-core` via `ansible`:
+  * #44 Moderate, affects versions < 2.17.6, ansible-core Incorrect Authorization vulnerability Moderate
+  * #47 Low, affects versions < 2.17.7: Ansible-Core vulnerable to content protections bypass Low
+* Vulnerabilities in transitive testing dependency `tornado` version 6.4.1 via `pytest-check-links`,  `nbconvert`, `nbclient`, `jupyter-client`:
+  * #46 High: Tornado has an HTTP cookie parsing DoS vulnerability High
+
+Accepted vulnerabilities:
+
+* Vulnerabilities in transitive testing dependency `python-jose` version 3.3.0 via `localstack` as there is no newer version available.
+  * #31 Critical: python-jose algorithm confusion with OpenSSH ECDSA keys Critical
+  * #32 Moderate: python-jose denial of service via compressed JWE content Moderate
+* Vulnerabilities in transitive dependency `ansible-core` 2.17.7 version via `ansible` as there is no newer version available.
+  * #43 High: Ansible vulnerable to Insertion of Sensitive Information into Log File High
+
+## Security Issues
+
+* #346: Dependency upgrade
+
 ## Refactorings
 
 * #333: Added project short tag in notebook tests
 * #339: Improved error reporting when the DockerDB doesn't start properly.
 
 ## Bug Fixes
 
- - #335: Fixed DNS resolution in ITDE when running jupyter notebook tests
- - #342: Updated the jupysql dependency to resolve the conflict with prettytable 
+* #335: Fixed DNS resolution in ITDE when running jupyter notebook tests
+* #342: Updated the jupysql dependency to resolve the conflict with prettytable

doc/developer_guide/dependencies.md

@@ -0,0 +1,52 @@
+# Update dependencies
+
+## Dependencies
+
+AI-Lab contains dependencies on multiple levels and specified in multiple places.
+
+* [pyproject.toml](https://github.com/exasol/ai-lab/blob/main/pyproject.toml) impacting [poetry.lock](https://github.com/exasol/ai-lab/blob/main/poetry.lock)
+* Requirements files in ansible scripts
+  * [jupyter_requirements.txt](https://github.com/exasol/ai-lab/blob/main/exasol/ds/sandbox/runtime/ansible/roles/jupyter/files/jupyter_requirements.txt)
+  * [notebook_requirements.txt](https://github.com/exasol/ai-lab/blob/main/exasol/ds/sandbox/runtime/ansible/roles/jupyter/files/notebook_requirements.txt)
+  * Including the notebook-connector and its dependencies SLCT and [slct_manager.py](https://github.com/exasol/notebook-connector/blob/main/exasol/nb_connector/slct_manager.py)
+* Dependencies in other ansible scripts, e.g.
+  * [docker/defaults/main.yml](https://github.com/exasol/ai-lab/blob/main/exasol/ds/sandbox/runtime/ansible/roles/docker/defaults/main.yml)
+  * [roles/jupyter/defaults/main.yml](https://github.com/exasol/ai-lab/blob/main/exasol/ds/sandbox/runtime/ansible/roles/jupyter/defaults/main.yml)
+* AMI base image, see [exasol/ds/sandbox/lib/config.py](https://github.com/exasol/ai-lab/blob/main/exasol/ds/sandbox/lib/config.py)
+* [test_dependencies.txt](https://github.com/exasol/ai-lab/blob/main/test/notebooks/test_dependencies.txt): Dependencies of the notebook tests
+* GitHub Workflows: no actual dependencies
+
+## Ansible packages
+
+The packages to be installed by Ansible are using pinned versions, e.g. for [docker](../../exasol/ds/sandbox/runtime/ansible/roles/docker/defaults/main.yml).
+
+In case ansible reports "no available installation candidate" for a specific version of a package, please search for newer versions of the package on https://packages.ubuntu.com/ or https://www.ubuntuupdates.org/.
+
+On `ubuntuupdates.org` you can use the [Package Search](https://www.ubuntuupdates.org/package_metas), please only use button "Package Search" or a URL like https://www.ubuntuupdates.org/package_metas?exact_match=1&q=network-manager.
+
+If the update is very new and not yet displayed on packages.ubuntu.com you can use
+
+```shell
+sudo apt-get update
+sudo apt search <package name>
+sudo apt-get install <package name>=<version>
+```
+
+Maybe installing the command [chdist](https://manpages.ubuntu.com/manpages/xenial/en/man1/chdist.1.html) could also be helpful, as it allows searching for packages and updates in other versions and distributions of ubuntu than the one installed on your local system.
+
+### Find Packages in All Ansible Scripts
+
+Shell function to find all packages in the ansible scripts
+
+```shell
+function xail-ansible-dependencies() {
+    local DIR=exasol/ds/sandbox/runtime/ansible/roles
+    for i in $( find "$DIR" -name "*.yml"); do
+        local DEPS=$(grep -E "[a-z]+=[0-9]" "$i")
+        if [ -n "$DEPS" ]; then
+            local LABEL=$(echo $i | sed -e "s|$DIR/||")
+            echo -e "\n$LABEL:\n$DEPS"
+        fi
+    done
+}
+```

doc/developer_guide/developer_guide.md

@@ -14,7 +14,7 @@ the virtual image formats.
 4. [Command Line Usage](commands.md)
 5. [Testing](testing.md)
 6. [Running tests in the CI](ci.md)
-7. [Updating Packages](updating_packages.md)
+7. [Dependencies and Updating Packages](dependencies.md)
 8. [Notebooks](notebooks.md)
 
-
+Section [Dependencies](dependencies.md) is dedicated to enumerating all places defining dependencies as the AI-Lab contains dependencies on multiple levels and specified in multiple places.

doc/developer_guide/updating_packages.md

@@ -14,4 +14,4 @@ jupyterlab_notebook_folder: "{{ user_home }}/notebooks"
 
 apt_dependencies:
   - virtualenv=20.13.0+ds-2
-  - git=1:2.34.1-1ubuntu1.11
+  - git=1:2.34.1-1ubuntu1.12

exasol/ds/sandbox/runtime/ansible/roles/jupyter/defaults/main.yml

@@ -1,8 +1,8 @@
-scikit-learn==1.5.1 # required for notebook sklearn
-matplotlib==3.7.4 # required for notebook sklearn
-jupysql==0.10.16 # required for multiple notebooks
+scikit-learn==1.6.1 # required for notebook sklearn
+matplotlib==3.10.0 # required for notebook sklearn
+jupysql==0.10.17 # required for multiple notebooks
 stopwatch.py>=2.0.1 # also required by ITDE
-exasol-notebook-connector==0.3.0
+exasol-notebook-connector==0.4.0
 pickleshare==0.7.5 # See https://github.com/exasol/ai-lab/issues/291 for details.
 ipyfilechooser==0.6.0 # required for SLC notebooks
-ipywidgets==8.1.1 # enable interactive Javascript widgets in the notebooks
+ipywidgets==8.1.5 # enable interactive Javascript widgets in the notebooks

exasol/ds/sandbox/runtime/ansible/roles/jupyter/files/notebook_requirements.txt

@@ -1,4 +1,4 @@
 ---
 
 apt_dependencies:
-  - rsync=3.2.7-0ubuntu0.22.04.2
+  - rsync=3.2.7-0ubuntu0.22.04.3