Detectron2 multimachine run stuck on spawn call

[unrue]

Dear Detectron2 user,

I'm using detectron2 on HPC multinode machine. When my test run on single node it works, but on two nodes the code stuck on spawn call, in detectron2/engine/launch.py:

`mp.spawn(
            _distributed_worker,
            nprocs=num_gpus_per_machine,
            args=(
                main_func,
                world_size,
                num_gpus_per_machine,
                machine_rank,
                dist_url,
                args,
                timeout,
            ),
            daemon=False,
        )

`

The following is how I set my code:

if __name__ == "__main__":

  import os
  import hostlist
  import dns
  import dns.resolver

  # get SLURM variables
  rank = int(os.environ['SLURM_PROCID'])
  local_rank = int(os.environ['SLURM_LOCALID'])
  size = int(os.environ['SLURM_NTASKS'])
  cpus_per_task = int(os.environ['SLURM_CPUS_PER_TASK'])

  # get node list from slurm
  hostnames = hostlist.expand_hostlist(os.environ['SLURM_JOB_NODELIST'])

  # get IDs of reserved GPU
  gpu_ids = os.environ['SLURM_STEP_GPUS'].split(",")

  # define MASTER_ADD & MASTER_PORT
  os.environ['MASTER_ADDR'] = hostnames[0]
  os.environ['MASTER_PORT'] = str(61000 + int(min(gpu_ids))) # to avoid port conflict on the same node

  print("MASTER ADDRESS: ", os.environ['MASTER_ADDR'])
  print("MASTER PORT: ", os.environ['MASTER_PORT'])

  suffix_name = ".m100.cineca.it"
  result = dns.resolver.resolve(hostnames[0]+suffix_name, 'A')

  master_ip = ''
  for ipval in result:
     master_ip = ipval.to_text()
     print('IP', master_ip)

  master_port = os.environ['MASTER_PORT']

  print('IP', master_ip)

  launch(main,
         num_machines = 2,
         num_gpus_per_machine = 1,
         machine_rank = 0,
         dist_url = 'tcp://' + master_ip + ':' + master_port)

   ....

And the following is my Slurm script`:

!/bin/bash

#SBATCH --time=00:10:00
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=8
#SBATCH --exclusive
#SBATCH -N 2
#SBATCH --gres=gpu:1

srun python3.8 ./train_multinode.py

Is there a particular reason why the spawn call stuck? I tried all available network interfaces on the machine, same behaviour. Thanks.

I'm using:

• python3.8
• NCCL version 2.11.4+cuda11.0
• detectron2 version 0.6
• pytorch 1.10.0