Issue showed in training panorama generation module #54
Description
Dear sir, I wonder how to fix the issue I met when running sh train_pano.sh?
(SGDreamer) sunzhenyu@mercury:~/Projects/SGDreamer/MVGen$ sh scripts/train_pano.sh
/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/plugins/training_type/ddp.py:68: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.
from torch.distributed.optim import DistributedOptimizer
Training dataset size: 9820
Validation dataset size: 1092
/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/core/lightning.py:2058: DeprecationWarning: `torch.distributed._sharded_tensor` will be deprecated, use `torch.distributed._shard.sharded_tensor` instead
from torch.distributed._sharded_tensor import pre_load_state_dict_hook, state_dict_hook
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/plugins/training_type/ddp.py:68: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.
from torch.distributed.optim import DistributedOptimizer
Training dataset size: 9820
Validation dataset size: 1092
/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/core/lightning.py:2058: DeprecationWarning: `torch.distributed._sharded_tensor` will be deprecated, use `torch.distributed._shard.sharded_tensor` instead
from torch.distributed._sharded_tensor import pre_load_state_dict_hook, state_dict_hook
/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/plugins/training_type/ddp.py:68: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.
from torch.distributed.optim import DistributedOptimizer
Training dataset size: 9820
Validation dataset size: 1092
/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/core/lightning.py:2058: DeprecationWarning: `torch.distributed._sharded_tensor` will be deprecated, use `torch.distributed._shard.sharded_tensor` instead
from torch.distributed._sharded_tensor import pre_load_state_dict_hook, state_dict_hook
initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/4
initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/4
/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/plugins/training_type/ddp.py:68: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.
from torch.distributed.optim import DistributedOptimizer
Training dataset size: 9820
Validation dataset size: 1092
/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/core/lightning.py:2058: DeprecationWarning: `torch.distributed._sharded_tensor` will be deprecated, use `torch.distributed._shard.sharded_tensor` instead
from torch.distributed._sharded_tensor import pre_load_state_dict_hook, state_dict_hook
initializing distributed: GLOBAL_RANK: 2, MEMBER: 3/4
initializing distributed: GLOBAL_RANK: 3, MEMBER: 4/4
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 4 processes
----------------------------------------------------------------------------------------------------
LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
| Name | Type | Params
-----------------------------------------------------
0 | text_encoder | CLIPTextModel | 340 M
1 | vae | AutoencoderKL | 83.7 M
2 | mv_base_model | MultiViewBaseModel | 1.0 B
-----------------------------------------------------
1.4 B Trainable params
0 Non-trainable params
1.4 B Total params
5,750.100 Total estimated model params size (MB)
Validation sanity check: 0it [00:00, ?it/s]/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/data_loading.py:110: UserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 128 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
rank_zero_warn(
[ WARN:0@21.280] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/8138835f78064c28b6c9c35a89eda7d6_skybox0_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@21.280] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/8138835f78064c28b6c9c35a89eda7d6_skybox1_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@21.280] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/8138835f78064c28b6c9c35a89eda7d6_skybox2_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@21.280] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/8138835f78064c28b6c9c35a89eda7d6_skybox3_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@21.280] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/8138835f78064c28b6c9c35a89eda7d6_skybox4_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@21.280] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/8138835f78064c28b6c9c35a89eda7d6_skybox5_sami.jpg'): can't open/read file: check file path/integrity
Validation sanity check: 0%| | 0/1 [00:00<?, ?it/s][ WARN:0@33.397] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/8112f9278bdf4dfbad53e140b846620c_skybox0_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@33.397] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/8112f9278bdf4dfbad53e140b846620c_skybox1_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@33.397] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/8112f9278bdf4dfbad53e140b846620c_skybox2_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@33.398] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/8112f9278bdf4dfbad53e140b846620c_skybox3_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@33.398] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/8112f9278bdf4dfbad53e140b846620c_skybox4_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@33.398] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/8112f9278bdf4dfbad53e140b846620c_skybox5_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@13.369] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/828c1d66f183441db9ac8bfcc7e5de31_skybox0_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@13.369] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/828c1d66f183441db9ac8bfcc7e5de31_skybox1_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@13.369] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/828c1d66f183441db9ac8bfcc7e5de31_skybox2_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@13.369] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/828c1d66f183441db9ac8bfcc7e5de31_skybox3_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@13.369] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/828c1d66f183441db9ac8bfcc7e5de31_skybox4_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@13.369] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/828c1d66f183441db9ac8bfcc7e5de31_skybox5_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@16.753] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/820774305e6f432ea789534d9cda5671_skybox0_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@16.753] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/820774305e6f432ea789534d9cda5671_skybox1_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@16.753] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/820774305e6f432ea789534d9cda5671_skybox2_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@16.753] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/820774305e6f432ea789534d9cda5671_skybox3_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@16.753] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/820774305e6f432ea789534d9cda5671_skybox4_sami.jpg'): can't open/read file: check file path/integrity
[ WARN:0@16.753] global loadsave.cpp:244 findDecoder imread_('data/mp3d_skybox/sT4fr6TAbpF/matterport_skybox_images/820774305e6f432ea789534d9cda5671_skybox5_sami.jpg'): can't open/read file: check file path/integrity
[rank1]: Traceback (most recent call last):
[rank1]: File "/home/sunzhenyu/Projects/SGDreamer/MVGen/train.py", line 79, in <module>
[rank1]: trainer.fit(model, train_loader, val_loader)
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in fit
[rank1]: self._call_and_handle_interrupt(
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 682, in _call_and_handle_interrupt
[rank1]: return trainer_fn(*args, **kwargs)
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in _fit_impl
[rank1]: self._run(model, ckpt_path=ckpt_path)
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1193, in _run
[rank1]: self._dispatch()
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1272, in _dispatch
[rank1]: self.training_type_plugin.start_training(self)
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
[rank1]: self._results = trainer.run_stage()
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1282, in run_stage
[rank1]: return self._run_train()
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1304, in _run_train
[rank1]: self._run_sanity_check(self.lightning_module)
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1368, in _run_sanity_check
[rank1]: self._evaluation_loop.run()
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/base.py", line 145, in run
[rank1]: self.advance(*args, **kwargs)
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 109, in advance
[rank1]: dl_outputs = self.epoch_loop.run(dataloader, dataloader_idx, dl_max_batches, self.num_dataloaders)
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/base.py", line 140, in run
[rank1]: self.on_run_start(*args, **kwargs)
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 87, in on_run_start
[rank1]: self._dataloader_iter = _update_dataloader_iter(data_fetcher, self.batch_progress.current.ready)
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 121, in _update_dataloader_iter
[rank1]: dataloader_iter = enumerate(data_fetcher, batch_idx)
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py", line 199, in __iter__
[rank1]: self.prefetching(self.prefetch_batches)
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py", line 258, in prefetching
[rank1]: self._fetch_next_batch()
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py", line 300, in _fetch_next_batch
[rank1]: batch = next(self.dataloader_iter)
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
[rank1]: data = self._next_data()
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 673, in _next_data
[rank1]: data = self._dataset_fetcher.fetch(index) # may raise StopIteration
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
[rank1]: data = [self.dataset[idx] for idx in possibly_batched_index]
[rank1]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in <listcomp>
[rank1]: data = [self.dataset[idx] for idx in possibly_batched_index]
[rank1]: File "/home/sunzhenyu/Projects/SGDreamer/MVGen/src/dataset/Matterport3D.py", line 77, in __getitem__
[rank1]: images_raw = [cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
[rank1]: File "/home/sunzhenyu/Projects/SGDreamer/MVGen/src/dataset/Matterport3D.py", line 77, in <listcomp>
[rank1]: images_raw = [cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
[rank1]: cv2.error: OpenCV(4.7.0) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/sunzhenyu/Projects/SGDreamer/MVGen/./train.py", line 79, in <module>
[rank0]: trainer.fit(model, train_loader, val_loader)
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in fit
[rank0]: self._call_and_handle_interrupt(
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 682, in _call_and_handle_interrupt
[rank0]: return trainer_fn(*args, **kwargs)
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in _fit_impl
[rank0]: self._run(model, ckpt_path=ckpt_path)
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1193, in _run
[rank0]: self._dispatch()
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1272, in _dispatch
[rank0]: self.training_type_plugin.start_training(self)
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
[rank0]: self._results = trainer.run_stage()
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1282, in run_stage
[rank0]: return self._run_train()
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1304, in _run_train
[rank0]: self._run_sanity_check(self.lightning_module)
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1368, in _run_sanity_check
[rank0]: self._evaluation_loop.run()
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/base.py", line 145, in run
[rank0]: self.advance(*args, **kwargs)
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 109, in advance
[rank0]: dl_outputs = self.epoch_loop.run(dataloader, dataloader_idx, dl_max_batches, self.num_dataloaders)
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/base.py", line 140, in run
[rank0]: self.on_run_start(*args, **kwargs)
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 87, in on_run_start
[rank0]: self._dataloader_iter = _update_dataloader_iter(data_fetcher, self.batch_progress.current.ready)
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 121, in _update_dataloader_iter
[rank0]: dataloader_iter = enumerate(data_fetcher, batch_idx)
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py", line 199, in __iter__
[rank0]: self.prefetching(self.prefetch_batches)
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py", line 258, in prefetching
[rank0]: self._fetch_next_batch()
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py", line 300, in _fetch_next_batch
[rank0]: batch = next(self.dataloader_iter)
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
[rank0]: data = self._next_data()
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 673, in _next_data
[rank0]: data = self._dataset_fetcher.fetch(index) # may raise StopIteration
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
[rank0]: data = [self.dataset[idx] for idx in possibly_batched_index]
[rank0]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in <listcomp>
[rank0]: data = [self.dataset[idx] for idx in possibly_batched_index]
[rank0]: File "/home/sunzhenyu/Projects/SGDreamer/MVGen/src/dataset/Matterport3D.py", line 77, in __getitem__
[rank0]: images_raw = [cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
[rank0]: File "/home/sunzhenyu/Projects/SGDreamer/MVGen/src/dataset/Matterport3D.py", line 77, in <listcomp>
[rank0]: images_raw = [cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
[rank0]: cv2.error: OpenCV(4.7.0) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
[rank3]: Traceback (most recent call last):
[rank3]: File "/home/sunzhenyu/Projects/SGDreamer/MVGen/train.py", line 79, in <module>
[rank3]: trainer.fit(model, train_loader, val_loader)
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in fit
[rank3]: self._call_and_handle_interrupt(
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 682, in _call_and_handle_interrupt
[rank3]: return trainer_fn(*args, **kwargs)
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in _fit_impl
[rank3]: self._run(model, ckpt_path=ckpt_path)
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1193, in _run
[rank3]: self._dispatch()
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1272, in _dispatch
[rank3]: self.training_type_plugin.start_training(self)
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
[rank3]: self._results = trainer.run_stage()
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1282, in run_stage
[rank3]: return self._run_train()
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1304, in _run_train
[rank3]: self._run_sanity_check(self.lightning_module)
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1368, in _run_sanity_check
[rank3]: self._evaluation_loop.run()
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/base.py", line 145, in run
[rank3]: self.advance(*args, **kwargs)
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 109, in advance
[rank3]: dl_outputs = self.epoch_loop.run(dataloader, dataloader_idx, dl_max_batches, self.num_dataloaders)
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/base.py", line 140, in run
[rank3]: self.on_run_start(*args, **kwargs)
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 87, in on_run_start
[rank3]: self._dataloader_iter = _update_dataloader_iter(data_fetcher, self.batch_progress.current.ready)
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 121, in _update_dataloader_iter
[rank3]: dataloader_iter = enumerate(data_fetcher, batch_idx)
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py", line 199, in __iter__
[rank3]: self.prefetching(self.prefetch_batches)
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py", line 258, in prefetching
[rank3]: self._fetch_next_batch()
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py", line 300, in _fetch_next_batch
[rank3]: batch = next(self.dataloader_iter)
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
[rank3]: data = self._next_data()
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 673, in _next_data
[rank3]: data = self._dataset_fetcher.fetch(index) # may raise StopIteration
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
[rank3]: data = [self.dataset[idx] for idx in possibly_batched_index]
[rank3]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in <listcomp>
[rank3]: data = [self.dataset[idx] for idx in possibly_batched_index]
[rank3]: File "/home/sunzhenyu/Projects/SGDreamer/MVGen/src/dataset/Matterport3D.py", line 77, in __getitem__
[rank3]: images_raw = [cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
[rank3]: File "/home/sunzhenyu/Projects/SGDreamer/MVGen/src/dataset/Matterport3D.py", line 77, in <listcomp>
[rank3]: images_raw = [cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
[rank3]: cv2.error: OpenCV(4.7.0) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
[rank2]: Traceback (most recent call last):
[rank2]: File "/home/sunzhenyu/Projects/SGDreamer/MVGen/train.py", line 79, in <module>
[rank2]: trainer.fit(model, train_loader, val_loader)
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in fit
[rank2]: self._call_and_handle_interrupt(
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 682, in _call_and_handle_interrupt
[rank2]: return trainer_fn(*args, **kwargs)
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in _fit_impl
[rank2]: self._run(model, ckpt_path=ckpt_path)
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1193, in _run
[rank2]: self._dispatch()
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1272, in _dispatch
[rank2]: self.training_type_plugin.start_training(self)
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
[rank2]: self._results = trainer.run_stage()
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1282, in run_stage
[rank2]: return self._run_train()
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1304, in _run_train
[rank2]: self._run_sanity_check(self.lightning_module)
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1368, in _run_sanity_check
[rank2]: self._evaluation_loop.run()
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/base.py", line 145, in run
[rank2]: self.advance(*args, **kwargs)
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 109, in advance
[rank2]: dl_outputs = self.epoch_loop.run(dataloader, dataloader_idx, dl_max_batches, self.num_dataloaders)
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/base.py", line 140, in run
[rank2]: self.on_run_start(*args, **kwargs)
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 87, in on_run_start
[rank2]: self._dataloader_iter = _update_dataloader_iter(data_fetcher, self.batch_progress.current.ready)
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 121, in _update_dataloader_iter
[rank2]: dataloader_iter = enumerate(data_fetcher, batch_idx)
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py", line 199, in __iter__
[rank2]: self.prefetching(self.prefetch_batches)
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py", line 258, in prefetching
[rank2]: self._fetch_next_batch()
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py", line 300, in _fetch_next_batch
[rank2]: batch = next(self.dataloader_iter)
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
[rank2]: data = self._next_data()
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 673, in _next_data
[rank2]: data = self._dataset_fetcher.fetch(index) # may raise StopIteration
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
[rank2]: data = [self.dataset[idx] for idx in possibly_batched_index]
[rank2]: File "/home/sunzhenyu/anaconda3/envs/SGDreamer/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in <listcomp>
[rank2]: data = [self.dataset[idx] for idx in possibly_batched_index]
[rank2]: File "/home/sunzhenyu/Projects/SGDreamer/MVGen/src/dataset/Matterport3D.py", line 77, in __getitem__
[rank2]: images_raw = [cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
[rank2]: File "/home/sunzhenyu/Projects/SGDreamer/MVGen/src/dataset/Matterport3D.py", line 77, in <listcomp>
[rank2]: images_raw = [cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
[rank2]: cv2.error: OpenCV(4.7.0) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
[rank0]:[W1022 15:36:14.488048791 ProcessGroupNCCL.cpp:1168] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())