When trying to scale to two nodes, I get "Error ignored in is_in_the_same_node: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:133] "

[carljones3000]

Hi I'm try to test scaling to two nodes.

I have output from two tests here, both of them lead to the "Error ignored in is_in_the_same_node: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:133] "

Test 1:
This is trying to run on two nodes, 4 3090 GPU's each, 8 GPU's total.
python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen1.5-32B-Chat --tensor-parallel-size 8 --max-model-len 12591 --swap-space 0

Test 2:
This is trying to run on two nodes, 1 3090 GPU each, 2 GPU's total.
python -m vllm.entrypoints.openai.api_server --model facebook/opt-13b --tensor-parallel-size 2

Any help would be appreciated.
-Carl

Test 1:

(vllmenv) x3@x:~$ python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen1.5-32B-Chat --tensor-parallel-size 8 --max-model-len 12591 --swap-space 0
INFO 07-19 04:03:47 api_server.py:206] vLLM API server version 0.5.1
INFO 07-19 04:03:47 api_server.py:207] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='Qwen/Qwen1.5-32B-Chat', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=12591, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=8, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=False, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, model_loader_extra_config=None, preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None)
INFO 07-19 04:03:47 config.py:698] Defaulting to use ray for distributed inference
2024-07-19 04:03:47,422	INFO worker.py:1586 -- Connecting to existing Ray cluster at address: 192.168.50.95:6379...
2024-07-19 04:03:47,428	INFO worker.py:1771 -- Connected to Ray cluster.
INFO 07-19 04:03:47 llm_engine.py:169] Initializing an LLM engine (v0.5.1) with config: model='Qwen/Qwen1.5-32B-Chat', speculative_config=None, tokenizer='Qwen/Qwen1.5-32B-Chat', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=12591, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=Qwen/Qwen1.5-32B-Chat, use_v2_block_manager=False, enable_prefix_caching=False)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
ERROR 07-19 04:34:07 parallel_state.py:1044] Error ignored in is_in_the_same_node: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:133] Timed out waiting 1800000ms for send operation to complete
ERROR 07-19 04:34:07 worker_base.py:348] Error executing method init_device. This might cause deadlock in distributed execution.
ERROR 07-19 04:34:07 worker_base.py:348] Traceback (most recent call last):
ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 340, in execute_method
ERROR 07-19 04:34:07 worker_base.py:348]     return executor(*args, **kwargs)
ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 126, in init_device
ERROR 07-19 04:34:07 worker_base.py:348]     init_worker_distributed_environment(self.parallel_config, self.rank,
ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 324, in init_worker_distributed_environment
ERROR 07-19 04:34:07 worker_base.py:348]     init_distributed_environment(parallel_config.world_size, rank,
ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 826, in init_distributed_environment
ERROR 07-19 04:34:07 worker_base.py:348]     _WORLD = init_world_group(ranks, local_rank, backend)
ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 713, in init_world_group
ERROR 07-19 04:34:07 worker_base.py:348]     return GroupCoordinator(
ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 195, in __init__
ERROR 07-19 04:34:07 worker_base.py:348]     if self.world_size > 1 and is_in_the_same_node(self.cpu_group):
ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 1049, in is_in_the_same_node
ERROR 07-19 04:34:07 worker_base.py:348]     torch.distributed.barrier(group=pg)
ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
ERROR 07-19 04:34:07 worker_base.py:348]     return func(*args, **kwargs)
ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3690, in barrier
ERROR 07-19 04:34:07 worker_base.py:348]     work.wait()
ERROR 07-19 04:34:07 worker_base.py:348] RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [192.168.50.95]:15274
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]:     return _run_code(code, main_globals, None,
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 216, in <module>
[rank0]:     engine = AsyncLLMEngine.from_engine_args(
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 431, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 360, in __init__
[rank0]:     self.engine = self._init_engine(*args, **kwargs)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 507, in _init_engine
[rank0]:     return engine_class(*args, **kwargs)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 243, in __init__
[rank0]:     self.model_executor = executor_class(
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 345, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 25, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 128, in __init__
[rank0]:     super().__init__(model_config, cache_config, parallel_config,
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 42, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 41, in _init_executor
[rank0]:     self._init_workers_ray(placement_group)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 191, in _init_workers_ray
[rank0]:     self._run_workers("init_device")
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 291, in _run_workers
[rank0]:     driver_worker_output = self.driver_worker.execute_method(
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 349, in execute_method
[rank0]:     raise e
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 340, in execute_method
[rank0]:     return executor(*args, **kwargs)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 126, in init_device
[rank0]:     init_worker_distributed_environment(self.parallel_config, self.rank,
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 324, in init_worker_distributed_environment
[rank0]:     init_distributed_environment(parallel_config.world_size, rank,
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 826, in init_distributed_environment
[rank0]:     _WORLD = init_world_group(ranks, local_rank, backend)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 713, in init_world_group
[rank0]:     return GroupCoordinator(
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 195, in __init__
[rank0]:     if self.world_size > 1 and is_in_the_same_node(self.cpu_group):
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 1049, in is_in_the_same_node
[rank0]:     torch.distributed.barrier(group=pg)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3690, in barrier
[rank0]:     work.wait()
[rank0]: RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [192.168.50.95]:15274
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 parallel_state.py:1044] Error ignored in is_in_the_same_node: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 1800000ms for recv operation to complete
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348] Error executing method init_device. This might cause deadlock in distributed execution.
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348] Traceback (most recent call last):
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 340, in execute_method
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]     return executor(*args, **kwargs)
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 126, in init_device
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]     init_worker_distributed_environment(self.parallel_config, self.rank,
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 324, in init_worker_distributed_environment
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]     init_distributed_environment(parallel_config.world_size, rank,
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 826, in init_distributed_environment
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]     _WORLD = init_world_group(ranks, local_rank, backend)
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 713, in init_world_group
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]     return GroupCoordinator(
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 195, in __init__
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]     if self.world_size > 1 and is_in_the_same_node(self.cpu_group):
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 1049, in is_in_the_same_node
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]     torch.distributed.barrier(group=pg)
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]     return func(*args, **kwargs)
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3690, in barrier
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348]     work.wait()
(RayWorkerWrapper pid=7429) ERROR 07-19 04:34:07 worker_base.py:348] RuntimeError: Application timeout caused pair closure
(RayWorkerWrapper pid=4200, ip=192.168.50.229) [rank1]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 923, in ensure_model_parallel_initialized
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     initialize_model_parallel(tensor_model_parallel_size,
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 889, in initialize_model_parallel
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     _TP = init_model_parallel_group(group_ranks,
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 732, in init_model_parallel_group
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     cpu_group = torch.distributed.new_group(ranks, backend="gloo")
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     func_return = func(*args, **kwargs)
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3868, in new_group
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     return _new_group_with_tag(ranks, timeout, backend, pg_options, None, use_local_synchronization=use_local_synchronization)
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3939, in _new_group_with_tag
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     pg, pg_store = _new_process_group_helper(
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1509, in _new_process_group_helper
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     backend_class = ProcessGroupGloo(backend_prefix_store, group_rank, group_size, timeout=timeout)
(RayWorkerWrapper pid=4200, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340] torch.distributed.DistStoreError: Socket Timeout
(RayWorkerWrapper pid=7492) ERROR 07-19 04:34:07 parallel_state.py:1044] Error ignored in is_in_the_same_node: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 1800000ms for recv operation to complete [repeated 2x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340] Error executing method init_device. This might cause deadlock in distributed execution. [repeated 6x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340] Traceback (most recent call last): [repeated 6x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 332, in execute_method [repeated 6x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     return executor(*args, **kwargs) [repeated 6x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 132, in init_device [repeated 6x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     init_worker_distributed_environment(self.parallel_config, self.rank, [repeated 6x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 346, in init_worker_distributed_environment [repeated 6x across cluster]
(RayWorkerWrapper pid=7492) ERROR 07-19 04:34:07 worker_base.py:348]     init_distributed_environment(parallel_config.world_size, rank, [repeated 2x across cluster]
(RayWorkerWrapper pid=7492) ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 826, in init_distributed_environment [repeated 2x across cluster]
(RayWorkerWrapper pid=7492) ERROR 07-19 04:34:07 worker_base.py:348]     _WORLD = init_world_group(ranks, local_rank, backend) [repeated 2x across cluster]
(RayWorkerWrapper pid=7492) ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 713, in init_world_group [repeated 2x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     return GroupCoordinator( [repeated 6x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 149, in __init__ [repeated 6x across cluster]
(RayWorkerWrapper pid=7492) ERROR 07-19 04:34:07 worker_base.py:348]     if self.world_size > 1 and is_in_the_same_node(self.cpu_group): [repeated 2x across cluster]
(RayWorkerWrapper pid=7492) ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 1049, in is_in_the_same_node [repeated 2x across cluster]
(RayWorkerWrapper pid=7492) ERROR 07-19 04:34:07 worker_base.py:348]     torch.distributed.barrier(group=pg) [repeated 2x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 89, in wrapper [repeated 6x across cluster]
(RayWorkerWrapper pid=7492) ERROR 07-19 04:34:07 worker_base.py:348]     return func(*args, **kwargs) [repeated 2x across cluster]
(RayWorkerWrapper pid=7492) ERROR 07-19 04:34:07 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3690, in barrier [repeated 2x across cluster]
(RayWorkerWrapper pid=7492) ERROR 07-19 04:34:07 worker_base.py:348]     work.wait() [repeated 2x across cluster]
(RayWorkerWrapper pid=7492) ERROR 07-19 04:34:07 worker_base.py:348] RuntimeError: Application timeout caused pair closure [repeated 2x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size, [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 923, in ensure_model_parallel_initialized [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     initialize_model_parallel(tensor_model_parallel_size, [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 889, in initialize_model_parallel [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     _TP = init_model_parallel_group(group_ranks, [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 732, in init_model_parallel_group [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     cpu_group = torch.distributed.new_group(ranks, backend="gloo") [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     func_return = func(*args, **kwargs) [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3868, in new_group [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     return _new_group_with_tag(ranks, timeout, backend, pg_options, None, use_local_synchronization=use_local_synchronization) [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3939, in _new_group_with_tag [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     pg, pg_store = _new_process_group_helper( [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1509, in _new_process_group_helper [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340]     backend_class = ProcessGroupGloo(backend_prefix_store, group_rank, group_size, timeout=timeout) [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) ERROR 07-19 04:34:07 worker_base.py:340] torch.distributed.DistStoreError: Socket Timeout [repeated 3x across cluster]
(RayWorkerWrapper pid=4288, ip=192.168.50.229) [rank4]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success). [repeated 3x across cluster]
/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
(vllmenv) x3@x:~$ ^C

Test 2:

(vllmenv) x3@x:~/tests/pytorch_test$ python -m vllm.entrypoints.openai.api_server --model facebook/opt-13b --tensor-parallel-size 2
INFO 07-19 10:50:24 api_server.py:206] vLLM API server version 0.5.1
INFO 07-19 10:50:24 api_server.py:207] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='facebook/opt-13b', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=False, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, model_loader_extra_config=None, preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None)
INFO 07-19 10:50:25 config.py:698] Defaulting to use ray for distributed inference
2024-07-19 10:50:25,970	INFO worker.py:1586 -- Connecting to existing Ray cluster at address: 192.168.50.95:6379...
2024-07-19 10:50:25,975	INFO worker.py:1771 -- Connected to Ray cluster.
INFO 07-19 10:50:26 llm_engine.py:169] Initializing an LLM engine (v0.5.1) with config: model='facebook/opt-13b', speculative_config=None, tokenizer='facebook/opt-13b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=facebook/opt-13b, use_v2_block_manager=False, enable_prefix_caching=False)
ERROR 07-19 11:20:32 parallel_state.py:1044] Error ignored in is_in_the_same_node: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:133] Timed out waiting 1800000ms for send operation to complete
ERROR 07-19 11:20:32 worker_base.py:348] Error executing method init_device. This might cause deadlock in distributed execution.
ERROR 07-19 11:20:32 worker_base.py:348] Traceback (most recent call last):
ERROR 07-19 11:20:32 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 340, in execute_method
ERROR 07-19 11:20:32 worker_base.py:348]     return executor(*args, **kwargs)
ERROR 07-19 11:20:32 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 126, in init_device
ERROR 07-19 11:20:32 worker_base.py:348]     init_worker_distributed_environment(self.parallel_config, self.rank,
ERROR 07-19 11:20:32 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 324, in init_worker_distributed_environment
ERROR 07-19 11:20:32 worker_base.py:348]     init_distributed_environment(parallel_config.world_size, rank,
ERROR 07-19 11:20:32 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 826, in init_distributed_environment
ERROR 07-19 11:20:32 worker_base.py:348]     _WORLD = init_world_group(ranks, local_rank, backend)
ERROR 07-19 11:20:32 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 713, in init_world_group
ERROR 07-19 11:20:32 worker_base.py:348]     return GroupCoordinator(
ERROR 07-19 11:20:32 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 195, in __init__
ERROR 07-19 11:20:32 worker_base.py:348]     if self.world_size > 1 and is_in_the_same_node(self.cpu_group):
ERROR 07-19 11:20:32 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 1049, in is_in_the_same_node
ERROR 07-19 11:20:32 worker_base.py:348]     torch.distributed.barrier(group=pg)
ERROR 07-19 11:20:32 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
ERROR 07-19 11:20:32 worker_base.py:348]     return func(*args, **kwargs)
ERROR 07-19 11:20:32 worker_base.py:348]   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3690, in barrier
ERROR 07-19 11:20:32 worker_base.py:348]     work.wait()
ERROR 07-19 11:20:32 worker_base.py:348] RuntimeError: Application timeout caused pair closure
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]:     return _run_code(code, main_globals, None,
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 216, in <module>
[rank0]:     engine = AsyncLLMEngine.from_engine_args(
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 431, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 360, in __init__
[rank0]:     self.engine = self._init_engine(*args, **kwargs)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 507, in _init_engine
[rank0]:     return engine_class(*args, **kwargs)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 243, in __init__
[rank0]:     self.model_executor = executor_class(
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 345, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 25, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 128, in __init__
[rank0]:     super().__init__(model_config, cache_config, parallel_config,
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 42, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 41, in _init_executor
[rank0]:     self._init_workers_ray(placement_group)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 191, in _init_workers_ray
[rank0]:     self._run_workers("init_device")
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 291, in _run_workers
[rank0]:     driver_worker_output = self.driver_worker.execute_method(
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 349, in execute_method
[rank0]:     raise e
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 340, in execute_method
[rank0]:     return executor(*args, **kwargs)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 126, in init_device
[rank0]:     init_worker_distributed_environment(self.parallel_config, self.rank,
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 324, in init_worker_distributed_environment
[rank0]:     init_distributed_environment(parallel_config.world_size, rank,
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 826, in init_distributed_environment
[rank0]:     _WORLD = init_world_group(ranks, local_rank, backend)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 713, in init_world_group
[rank0]:     return GroupCoordinator(
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 195, in __init__
[rank0]:     if self.world_size > 1 and is_in_the_same_node(self.cpu_group):
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 1049, in is_in_the_same_node
[rank0]:     torch.distributed.barrier(group=pg)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3690, in barrier
[rank0]:     work.wait()
[rank0]: RuntimeError: Application timeout caused pair closure
(RayWorkerWrapper pid=38752, ip=192.168.50.229) [rank1]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340] Error executing method init_device. This might cause deadlock in distributed execution.
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340] Traceback (most recent call last):
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 332, in execute_method
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]     return executor(*args, **kwargs)
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 132, in init_device
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]     init_worker_distributed_environment(self.parallel_config, self.rank,
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 346, in init_worker_distributed_environment
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 923, in ensure_model_parallel_initialized
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]     initialize_model_parallel(tensor_model_parallel_size,
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 889, in initialize_model_parallel
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]     _TP = init_model_parallel_group(group_ranks,
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 732, in init_model_parallel_group
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]     return GroupCoordinator(
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 149, in __init__
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]     cpu_group = torch.distributed.new_group(ranks, backend="gloo")
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 89, in wrapper
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]     func_return = func(*args, **kwargs)
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3868, in new_group
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]     return _new_group_with_tag(ranks, timeout, backend, pg_options, None, use_local_synchronization=use_local_synchronization)
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3939, in _new_group_with_tag
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]     pg, pg_store = _new_process_group_helper(
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]   File "/home/user2/software/anaconda3/envs/vllmenv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1509, in _new_process_group_helper
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340]     backend_class = ProcessGroupGloo(backend_prefix_store, group_rank, group_size, timeout=timeout)
(RayWorkerWrapper pid=38752, ip=192.168.50.229) ERROR 07-19 11:20:32 worker_base.py:340] torch.distributed.DistStoreError: Socket Timeout
/home/user3/software/anaconda3/envs/vllmenv/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
(vllmenv) x3@x:~/tests/pytorch_test$