loading synthetic scenes in private_test_hard_two_stage split consumes too much memory.

[TikaToka]

Hello team.

I am trying to submit to leaderboard for challenge. However, loading synthetic scenes uses too much memory.

I am using PC that has 128gb ram, but when I try loading private_test_hard_two_stage split, original scenes are no problem, but loading synthetic scenes invokes overflow of memory.

Is there a way to load it more efficiently?

for more information, loading navhard_two_stage, which also need synthetic scenes, do not consumes memory a lot. while codebase for scene and sceneloader seems identical.

Thank you in advance!

[DanielDauner]

Hi @TikaToka,

Thanks for reaching out. May I ask what worker argument you are using? See also #83 #60 #32

Best,
Daniel

[TikaToka]

Hello @DanielDauner,
I am using a same config as default, so I assume it will be ray_distributed_no_torch.
I do understand that as this type of worker uses CPU for inference, it may use a lot of Memory.
However, navhard_two_stage split, which is more larger than private_test_hard_two_stage and also contains synthetic scenes, does not use that much of the memory while the model and the config are the same.

[DanielDauner]

Okay I see. Memory is always tricky with ray. A workaround would be to use worker=single_machine_thread_pool or even worker=sequential. You can also use ray and limit the number of parallel threads with worker.threads_per_node=... (i.e. CPU threads divided by 2 or 4).

I don't really know why ray is causing issues for navhard_two_stage (but not for private_test_hard_two_stage). Maybe navhard_two_stage has more synthetic key frames.

[TikaToka]

@DanielDauner Thank you for quick response. I also have tried to use only 1 worker, but it didn't helped, which is very annoying...

[TikaToka]

@DanielDauner Thank you for quick response.

Okay I see. Memory is always tricky with ray. A workaround would be to use worker=single_machine_thread_pool or even worker=sequential. You can also use ray and limit the number of parallel threads with worker.threads_per_node=... (i.e. CPU threads divided by 2 or 4).

I also have tried to use only 1 worker, but it didn't helped, which is very annoying...

I don't really know why ray is causing issues for navhard_two_stage (but not for private_test_hard_two_stage). Maybe navhard_two_stage has more synthetic key frames.

and It is actually the opposite, private_test_hard_two_stage is a problem, not a navhard_two_stage.

[DanielDauner]

Hey @TikaToka

sorry to hear that wasn't helpful. Could you verify that this is a general problem, e.g. by testing private_test_hard_two_stage with the baseline LTF agent? Have you tried worker=single_machine_thread_pool already?

Best,
Daniel

[TikaToka]

Hello, @DanielDauner.

Have you tried worker=single_machine_thread_pool already?

Yes, I've tried with max_workers=1 but still uses too much Memory, occuring OOM.

[rdesc]

I'm facing a similar issue

[rdesc]

Did you figure it out @TikaToka ?

[rdesc]

I'm also confused why navhard_two_stage works fine which has 450 Stage 1 and 5462 Stage 2 observations compared to the private_test_hard_two_stage with 140 Stage 1 and 1732 Stage 2 observations which does not work

[rdesc]

Ok solved the issue. I successfully loaded private_test_hard_two_stage by reducing the number of sensor info to load via the SensorConfig, e.g.

SensorConfig(
            cam_f0=[0,1,2,3],
            cam_l0=[0,1,2,3],
            cam_l1=False,
            cam_l2=False,
            cam_r0=[0,1,2,3],
            cam_r1=False, 
            cam_r2=False, 
            cam_b0=False, 
            lidar_pc=False,
        ) 

[TikaToka]

Ok solved the issue. I successfully loaded private_test_hard_two_stage by reducing the number of sensor info to load via the SensorConfig, e.g.

SensorConfig(
cam_f0=[0,1,2,3],
cam_l0=[0,1,2,3],
cam_l1=False,
cam_l2=False,
cam_r0=[0,1,2,3],
cam_r1=False,
cam_r2=False,
cam_b0=False,
lidar_pc=False,
)

Hello, @rdesc. This approach might be helpful. However, since I need all the sensors except LiDAR, it doesn’t fully address my case (only removing lidar do not help). Still, thank you very much for sharing!

[rdesc]

@TikaToka ah damn yea thats annoying, not sure then.. does it work if you just load lidar? Could maybe hack it so that you load the lidar / rgb in 2 separate steps

[TikaToka]

@rdesc Yeah, unfortunately that’s the case :(
The private datasets don’t include LiDAR sensors, since NAVSIM is designed to work only with RGB, so it is not applicable here. (Datasets directly from OpenScene such as navtrain do have lidars) So, I think there might be a some differences inside the library btw private and non_private maybe? but not sure bout it.

What I’m thinking is that, because the current implementation loads all sensor data at the beginning, it might be possible to modify the code to load sensor data on demand based on the token value instead.