All jobs fail if vault encryption keys are replaced.

[bernt-matthias]

Describe the bug

I accidentally deleted vault encrytion keys in my dev instance. I expected that just the secrets are not decryptable. Instead job submission broke completely:

galaxy.jobs.runners ERROR 2024-03-22 13:46:14,594 [pN:main.2,p:1158411,tN:SlurmRunner.work_thread-3] (unknown) Unhandled exception calling fail_job
Traceback (most recent call last):
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/jobs/runners/__init__.py", line 170, in run_next
    method(arg)
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/jobs/runners/__init__.py", line 586, in fail_job
    job_state.job_wrapper.fail(
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/jobs/__init__.py", line 1476, in fail
    self._fix_output_permissions()
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/jobs/__init__.py", line 1376, in _fix_output_permissions
    for path in [dp.real_path for dp in self.job_io.get_mutable_output_fnames()]:
                                        ^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/jobs/__init__.py", line 1084, in job_io
    file_sources_dict=self.app.file_sources.to_dict(for_serialization=True, user_context=user_context),
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/files/__init__.py", line 193, in to_dict
    "file_sources": self.plugins_to_dict(for_serialization=for_serialization, user_context=user_context),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/files/__init__.py", line 185, in plugins_to_dict
    el = file_source.to_dict(for_serialization=for_serialization, user_context=user_context)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/files/sources/__init__.py", line 354, in to_dict
    rval.update(self._serialization_props(user_context=user_context))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/files/sources/_pyfilesystem2.py", line 91, in _serialization_props
    effective_props[key] = self._evaluate_prop(val, user_context=user_context)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/files/sources/__init__.py", line 433, in _evaluate_prop
    rval = fill_template(prop_val, context=template_context, futurized=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/util/template.py", line 143, in fill_template
    raise first_exception or e
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/util/template.py", line 87, in fill_template
    return unicodify(t, log_exception=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/util/__init__.py", line 1153, in unicodify
    value = str(value)
            ^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/Cheetah/Template.py", line 1053, in __unicode__
    return getattr(self, mainMethName)()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cheetah_DynamicallyCompiledCheetahTemplate_1711111574_0387433_22521.py", line 86, in respond
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/security/vault.py", line 210, in read_secret
    return self.vault.read_secret(f"user/{self.user.id}/{key}")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/security/vault.py", line 247, in read_secret
    return self.vault.read_secret(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/security/vault.py", line 267, in read_secret
    return self.vault.read_secret(f"/{self.prefix}/{key}")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/security/vault.py", line 158, in read_secret
    return f.decrypt(key_obj.value.encode("utf-8")).decode("utf-8")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/cryptography/fernet.py", line 211, in decrypt
    raise InvalidToken
cryptography.fernet.InvalidToken

Galaxy Version and/or server at which you observed the bug
Galaxy Version: 23.2
Commit: 6f0711a

Browser and Operating System
Operating System: Rocky 9
Browser: Firefox

To Reproduce

• Setup vault
• Store some user secrets
• Replace encryption keys
• Start any job

Expected behavior

Unsure.

Maybe something should be sent to the logs. But jobs should run, or?
Or better: Maybe only jobs using something in the vault should fail.

[dannon]

Discussion from weekly dev meeting shouldn't actually fail until trying to access one of these missing parameters. at "file_sources": self.plugins_to_dict(for_serialization=for_serialization, user_context=user_context),, capture exception and keep in the dict to rethrow on the other end IIF required.

[ahmedhamidawan]

Bumping this to 25.1 if work on this will target the next release? We can adjust if that's not the case