Projection of predicted trajectory onto the front camera view

[bskkimm]

Hi, I wonder how the trajectory can be projected onto the front camera view. I know same visualization is possible in BEV case in the visualization tool you have created (add_trajectory_bev).

I was trying to do it, utilizing the extrinsic and intrinsic params of the camera in vain.

I somehow reached how the sensor_to_lidar (camera to lidar) extrinsic params can be extracted but unable to obtain local or global to camera extrinsic params as the trajectory output is the local ego coordinate.

In other words, convert local ego trajectories into the front camera coordinate and then project it onto the camera image plane using intrinsic params.

Addtional question:

Is there any approach that selectively accesses a certain type of scenario or scene such as challenging scenarios to evaluate how well the model navigates them.

Your advice would be very helpful!

[VVeiCao]

@mh0797 Can you help to answer this question?

[DanielDauner]

Hi @bskkimm,

Maybe have a look at navsim.visualization.camera.py. We have some scripts there to project the bounding box annotations or the LiDAR point cloud into the image. You might be able to re-use some of the functions for adding the trajectory onto the camera images. Thanks for you feedback. We might extend the visualization scripts in a future update.

For finding challenging scenarios, you might want to filter scenes where you planner fails. Similarly, in the NAVSIM paper, we filter for scenes where constant velocity fails. I hope that helps!

Best,
Daniel

[DanielDauner]

Closing this issue. Feel free to re-open, if you have further questions!

[maxiuw]

Hi I wrote the script to do that and opened a pull request #120