Implementing Raycasting for 3D Labeling/Annotation in Deck.gl Point Cloud Viewer

[iamravisingh]

Hi Deck.gl Team and Community,

We've developed a React-based web application that uses Deck.gl to render point cloud data, providing first-person and orbit view navigation. We're now looking to add 3D labeling/annotation functionality to enable tasks like segmenting objects (e.g., cropping pillars or columns) within the point clouds.

We understand that raycasting is a crucial technique for enabling interactive 3D selection and annotation. However, we haven't implemented raycasting in our application yet and are seeking guidance on how to get started with it within the Deck.gl context.

Our goal is to use raycasting to enable the following:

Bounding Box Creation (Raycasting-Assisted):

Enable users to define the corners of a 3D bounding box by clicking on points.
We need to understand how to translate these clicks into 3D coordinates and then visualize the resulting bounding box.

Cylinder Creation (Raycasting-Assisted):

Enable users to define cylinder parameters (base circle, height) by clicking on points.
We need to know how to get ray direction from click point, and then using that direction to correctly align cylinder.

Region/Volume Selection:

Explore how raycasting can be used to select a region of points or a specific 3D volume within the point cloud.

Performance Optimization:
We're working with potentially large point cloud datasets. We'd like to understand any Deck.gl-specific optimizations or techniques for efficient raycasting.
We need guidance on spatial indexing structures, that will improve performance.
Specifically, we'd appreciate guidance on:

How to implement raycasting in a Deck.gl application:

What are the recommended approaches and libraries for adding raycasting functionality?

Integrating raycasting with Deck.gl's PointCloudLayer:

How can we effectively use raycasting to interact with the rendered point cloud data?

Converting raycasting results into 3D coordinates:
`How can we translate the raycasting output (intersection points, distances) into the 3D coordinates of the selected points?

`Visualizing annotation shapes (bounding boxes, cylinders) in Deck.gl:

What are the recommended layers or techniques for rendering these shapes interactively?

Are there any examples or code snippets that demonstrate how to implement raycasting for point selection or 3D annotation in Deck.gl?
We are eager to learn how to implement raycasting and leverage its capabilities for 3D labeling in our application. Any help or suggestions from the Deck.gl team and community would be greatly appreciated.

Below are 2 different viewers being implemented i.e orbit and firstpersion view for your reference.

OrbitView

FirstpersoinView

Thank you for your time and expertise.