2025/05/14 Meeting Notes

[himanshunaidu]

Progress Update

Segmentation Post-Processing

1. Union of Masks

Implemented Union of Masks with 5 consecutive frames.
The ideal number of consecutive frames to use would depend upon the FPS. Since the current FPS is only 5, this current number of masks for union is not ideal, as we want only a few milliseconds worth of union.
We can think of a strategy of displaying only the 15 frames per second, but actually taking in all the 60 frames from the camera and passing the last few through the segmentation model once the capture is clicked, and then performing a union of masks. This will have a slight performance impact at the start of the AnnotationView, which should be reasonable depending on the segmentation model itself.

2. AnnotationView Post-Processing
   Implemented post-processing in Annotation View, which involves contour detection, polygon approximation and centroid calculation.
   We are still using only the centroid for Depth processing, which we would want to replace with trimmed mean.

3. Sidewalk Segmentation
   Implemented segmentation and mapping of way-type classes. Now can upload way-type classes to TDEI workspaces.

API Transmission

1. Changeset Lifecycle change
   Previously Changesets were automatically opened and closed within the ContentView, not giving the user much control over what is being uploaded.
   Now the opening and closing of the Changeset is done in the SetupView. While the opening is automatic, the user has the ability to close the changeset. This can eventually help even better control the lifecycle of the Changesets.

Other Fixes

1. Depth Calculation has been fixed to utilize the correct centroids of the segmentation masks.
   There was a bug with centroid calculation previously. Utilized shoelace method to fix the calculation.

2. AnnotationView can send multiple instances of the same class to TDEI Workspaces.

3. Performance Issues fixes.

Next Steps

Segmentation Post-Processing

1. Fix remaining GPS-related issues with object location calculation algorithm, as we fell back to centroid usage instead of trimmed mean.

2. Start utilizing other sensor-related information from the device for better calculation of sidewalk width

3. Start assessing the ideal bounds for sidewalks and other way types.
   This would ideally depend upon the segmentation model and where does it generally have the highest confidence for segmentation.

API Transmission

1. Decide on what would be the appropriate limit on the number of instances of a class that should be segmented
   The model could end up producing several separate instances of the class. We will need to restrict the number, based on size and total number (heuristic class-based thresholds).

2. Decide whether it is practical to have multiple instances for sidewalk and other way-type classes
   Currently, we only retain the largest sidewalk segment. We should assess on whether it is practical to support multiple instances.

Other Fixes

1. There are a couple of data race issues occurring while running the ContentView (and sometimes the AnnotationView). Need to fix these.

2. Take care of remaining orientation issues by using dynamic layout related SwiftUI classes such as Layout.

3. Streamline the code
   The code related to Segmentation post-processing and AnnotationView have become too bloated, giving lesser control over their lifecycles, thus may be leading to multiple re-renders.