OSError: [Errno 28] No space left on device #31
Description
I'm running into out of storage issues while training (the "PPO" algorithm) as follows:
2022-08-24 11:57:32,014 WARNING worker.py:1404 -- Traceback (most recent call last):
File "python/ray/_raylet.pyx", line 896, in ray._raylet.spill_objects_handler
File "python/ray/_raylet.pyx", line 899, in ray._raylet.spill_objects_handler
File "/home/xxx/anaconda3/envs/mujoco_env/lib/python3.7/site-packages/ray/external_storage.py", line 662, in spill_objects
return _external_storage.spill_objects(object_refs, owner_addresses)
File "/home/xxx/anaconda3/envs/mujoco_env/lib/python3.7/site-packages/ray/external_storage.py", line 303, in spill_objects
return self._write_multiple_objects(f, object_refs, owner_addresses, url)
File "/home/xxx/anaconda3/envs/mujoco_env/lib/python3.7/site-packages/ray/external_storage.py", line 151, in _write_multiple_objects
written_bytes = f.write(payload)
OSError: [Errno 28] No space left on device
An unexpected internal error occurred while the IO worker was spilling objects: [Errno 28] No space left on device
Approximately how much space is required for the "apex" project to train. thanks