Skip to content

train error : SystemError: (Fatal) Blocking queue is killed because the data reader raises an exception. #499

@bigyaofei

Description

@bigyaofei

W1011 17:07:55.921258 79684 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 12.2, Runtime API Version: 11.8
W1011 17:07:55.921286 79684 gpu_resources.cc:164] device: 0, cuDNN Version: 8.9.
Exception in thread Thread-2:
Traceback (most recent call last):
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/threading.py", line 980, in _bootstrap_inner
self.run()
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/threading.py", line 917, in run
self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 603, in _thread_loop
File "/home/fyao/prj/Paddle3D/tools/train.py", line 269, in
batch = self._get_data()
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 752, in _get_data
main(args)
File "/home/fyao/prj/Paddle3D/tools/train.py", line 264, in main
batch.reraise()
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/site-packages/paddle/io/dataloader/worker.py", line 187, in reraise
trainer.train()
File "/home/fyao/prj/Paddle3D/paddle3d/apis/trainer.py", line 327, in train
raise self.exc_type(msg)
TypeError: DataLoader worker(1) caught TypeError with message:
Traceback (most recent call last):
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/site-packages/paddle/io/dataloader/worker.py", line 372, in _worker_loop
batch = fetcher.fetch(indices)
File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/site-packages/paddle/io/dataloader/fetcher.py", line 77, in fetch
data.append(self.dataset[idx])
File "/home/fyao/prj/Paddle3D/paddle3d/datasets/kitti/kitti_pointcloud_det.py", line 62, in getitem
sample = self.transforms(sample)
File "/home/fyao/prj/Paddle3D/paddle3d/transforms/base.py", line 45, in call
sample = t(sample)
File "/home/fyao/prj/Paddle3D/paddle3d/transforms/sampling.py", line 278, in call
samples = self.sampling(sample, num_samples_per_class)
File "/home/fyao/prj/Paddle3D/paddle3d/transforms/sampling.py", line 205, in sampling
sampling_bboxes_3d, mv_height = self._put_boxes_on_road_planes(
File "/home/fyao/prj/Paddle3D/paddle3d/transforms/sampling.py", line 139, in _put_boxes_on_road_planes
a, b, c, d = road_planes
TypeError: cannot unpack non-iterable NoneType object
for sample in self.train_dataloader:

File "/home/fyao/anaconda3/envs/paddle/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 826, in next
self.reader.read_next_list()[0]
SystemError: (Fatal) Blocking queue is killed because the data reader raises an exception.
[Hint: Expected killed
!= true, but received killed_:1 == true:1.] (at /paddle/paddle/fluid/operators/reader/blocking_queue.h:175)

why?

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions