Skip to content

[feat] Enable launchers to forward environment variables #3464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion docs/config_reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -518,8 +518,10 @@ System Partition Configuration
The program will be launched locally.
- ``lrun``: Parallel programs will be launched using `LC Launcher <https://hpc.llnl.gov/training/tutorials/using-lcs-sierra-system#lrun>`__'s ``lrun`` command.
- ``lrun-gpu``: Parallel programs will be launched using `LC Launcher <https://hpc.llnl.gov/training/tutorials/using-lcs-sierra-system#lrun>`__'s ``lrun -M "-gpu"`` command that enables the CUDA-aware Spectrum MPI.
- ``mpirun``: Parallel programs will be launched using the ``mpirun`` command.
- ``mpiexec``: Parallel programs will be launched using the ``mpiexec`` command.
- ``mpirun``: Parallel programs will be launched using the (generic) ``mpirun`` command.
- ``mpirun-intelmpi``: Parallel programs will be launched using the Intel MPI's ``mpirun`` command.
- ``mpirun-openmpi``: Parallel programs will be launched using the OpenMPI's ``mpirun`` command.
- ``pdsh``: Parallel programs will be launched using the ``pdsh`` command. This launcher uses the partition's :attr:`~config.systems.partitions.access` property in order to determine the options to be passed to ``pdsh``.
- ``srun``: Parallel programs will be launched using `Slurm <https://slurm.schedmd.com/srun.html>`__'s ``srun`` command.
- ``srunalloc``: Parallel programs will be launched using `Slurm <https://slurm.schedmd.com/srun.html>`__'s ``srun`` command, but job allocation options will also be emitted.
Expand All @@ -542,6 +544,10 @@ System Partition Configuration
- ``upcrun``: Parallel programs will be launched using the `UPC <https://upc.lbl.gov/>`__ ``upcrun`` command.
- ``upcxx-run``: Parallel programs will be launched using the `UPC++ <https://bitbucket.org/berkeleylab/upcxx/wiki/Home>`__ ``upcxx-run`` command.

.. versionadded:: 4.9

The ``mpirun-intelmpi`` and ``mpirun-openmpi`` parallel launchers were added.

.. tip::

.. versionadded:: 4.0.0
Expand Down
17 changes: 17 additions & 0 deletions reframe/core/launchers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,22 @@ class JobLauncher(metaclass=_JobLauncherMeta):
#: :default: ``[]``
options = variable(typ.List[str], value=[])

#: Dictionary of environment variables to be passed via the job launcher
#: invocation. The keys are the variable names and the values are their
#: corresponding values.
#:
#: This is supported by the following launchers only:
#:
#: - ``srun``
#: - ``mpirun-openmpi``
#: - ``mpirun-intelmpi``
#:
#: :type: :class:`Dict[str, str]`
#: :default: ``{}``
#:
#: .. versionadded:: 4.9
env_vars = variable(typ.Dict[str, str], value={})

#: Optional modifier of the launcher command.
#:
#: This will be combined with the :attr:`modifier_options` and prepended to
Expand All @@ -59,6 +75,7 @@ class JobLauncher(metaclass=_JobLauncherMeta):

def __init__(self):
self.options = []
self.env_vars = {}

@abc.abstractmethod
def command(self, job):
Expand Down
22 changes: 22 additions & 0 deletions reframe/core/launchers/mpi.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ def command(self, job):
if self.use_cpus_per_task and job.num_cpus_per_task:
ret.append(f'--cpus-per-task={job.num_cpus_per_task}')

if self.env_vars:
env_vars = ','.join(f'{k}={v}' for k, v in self.env_vars.items())
ret.append(f'--export={env_vars}')

return ret


Expand Down Expand Up @@ -109,6 +113,24 @@ def command(self, job):
return ['mpirun', '-np', str(job.num_tasks)]


@register_launcher('mpirun-openmpi')
class MpirunOpenMPILauncher(JobLauncher):
def command(self, job):
cmd = ['mpirun', '-np', str(job.num_tasks)]
for name, value in self.env_vars.items():
cmd += ['-x', f'{name}={value}']
return cmd


@register_launcher('mpirun-intelmpi')
class MpirunIntelMPILauncher(JobLauncher):
def command(self, job):
cmd = ['mpirun', '-np', str(job.num_tasks)]
for name, value in self.env_vars.items():
cmd += ['-genv', name, value]
return cmd


@register_launcher('mpiexec')
class MpiexecLauncher(JobLauncher):
def command(self, job):
Expand Down
16 changes: 15 additions & 1 deletion unittests/test_launchers.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ def job(make_job, launcher):
job.exclusive_access = True
job.options = ['--gres=gpu:4', '#DW jobdw anything']
job.launcher.options = ['--foo']
job.launcher.env_vars = {'FOO': 'bar', 'BAR': 'baz'}
return job


Expand Down Expand Up @@ -127,7 +128,20 @@ def test_run_command(job):
elif launcher_name == 'mpirun':
assert command == 'mpirun -np 4 --foo'
elif launcher_name == 'srun':
assert command == 'srun --cpus-per-task=2 --foo'
assert command == ('srun '
'--cpus-per-task=2 '
'--export=FOO=bar,BAR=baz '
'--foo')
elif launcher_name == 'mpirun-openmpi':
assert command == ('mpirun -np 4 '
'-x FOO=bar '
'-x BAR=baz '
'--foo')
elif launcher_name == 'mpirun-intelmpi':
assert command == ('mpirun -np 4 '
'-genv FOO bar '
'-genv BAR baz '
'--foo')
elif launcher_name == 'srunalloc':
assert command == ('srun '
'--job-name=fake_job '
Expand Down
Loading