2025/05/21 Meeting Notes

[himanshunaidu]

Progress Update

Segmentation Post-Processing

1. Start integrating other sensor-related information for better post-processing. (IN PROGRESS)
   Trying to integrate ARKit, because it performs all the sensor-fusion under the hood.
   Could help improve even the transformation of consecutive frames for union of masks, as it uses VIO to help calculate relative transforms. It does not make the assumption of the consecutive frames being co-planar, unlike Homography transforms.
   It will also give other details such as the camera calibration matrix, which, combined with FOV details, could give us a more accurate estimation of details such as sidewalk width.

API Transmission

1. More streamlined self-contained updates to TDEI workspace
   For node-type classes, instead of calling the API once for every instance of an object, all the instances are now passed to the TDEI workspace.
   For way-type classes, instead of first passing the relevant node, then getting the updated node and then modifying the way accordingly, we now create the node and modify the way together.

Other Fixes

1. Streamline the code
   Code related to Segmentation post-processing and AnnotationView have been streamlined. This led to better control over the components' lifecycle, and helped prevent spurious re-renders.

2. Fixed orientation issues
   Using dynamic layout

3. Tackled a couple of data race issues occurring while running the ContentView (and sometimes the AnnotationView)
   Having a hard time reproducing these.
   While it is not completely confirmed on whether these issues have been eliminated, thanks to the code streamlining, they do not seem to be occurring as much. Especially the AnnotationView issues.
   Need to test them further.

Data Collection

1. Testing Grounded-SAM2 on its ability to help gather coarse annotations
   Could help scale up data collection for fine-tuning.

2. Explore data collection methods
   Some options:

• SideSeeing dataset (utilize an Android app called Multi-Sensor to collect consecutive image frames along with other data such as location, accelerometer, etc.) https://arxiv.org/abs/2407.06464
• LiDAR Depth-Map Capture For iOS: https://github.com/ioridev/LiDAR-Depth-Map-Capture-for-iOS
• Stray Scanner App: https://github.com/strayrobots/scanner?tab=readme-ov-file

Next Steps

Segmentation Post-Processing

1. Complete integration of other sensor-related information for better post-processing

Data Collection

1. Collect more data and test with Grounded-SAM2.