How is registration possible with non-overlapping sub-map start clouds? (Sec. 3.3)

[ShilangChen1011]

Description
In Section 3.3 you note that

“We found that directly registering Gaussian means from different agents is unreliable, as Gaussians representing overlapping regions can have widely varying distributions across agents. To address this, we anchor an input point cloud at the start of each sub-map and use it for registration.”

However, if two sub-maps’ starting point clouds have no shared field of view, it seems there would be no overlap to drive a reliable alignment.

Questions

1. Does your method implicitly assume that all agents begin mapping from the same physical location (thus guaranteeing some overlap in their initial point clouds)?
2. If not, what mechanism ensures sufficient correspondence between non‐overlapping start clouds so that registration can succeed?

Thanks for any clarification!

[VladimirYugay]

Hey there. Thanks for your interest in our work!

No, our method doesn't expect that overlap. If there is no overlap, the loop candidates will be empty because there are no visually similar images. And even if there are false positive candidates, the loops are filtered afterwards here.

[lemonci]

Description In Section 3.3 you note that

“We found that directly registering Gaussian means from different agents is unreliable, as Gaussians representing overlapping regions can have widely varying distributions across agents. To address this, we anchor an input point cloud at the start of each sub-map and use it for registration.”

However, if two sub-maps’ starting point clouds have no shared field of view, it seems there would be no overlap to drive a reliable alignment.

Questions

1. Does your method implicitly assume that all agents begin mapping from the same physical location (thus guaranteeing some overlap in their initial point clouds)?

2. If not, what mechanism ensures sufficient correspondence between non‐overlapping start clouds so that registration can succeed?

Thanks for any clarification!

Hi, though I am not the author, I have been playing on this project for a few weeks. So let me try to clarify a bit:

1. Unlike some other multi-agent SLAM works in which each agent only generates a single submap, in MAGiC-SLAM, each agent generates multiple submaps and saves (1)some meta information (including the feature vectors of the first keyframe of the submap), (2)the gaussian model parameters and (3)the point cloud of the submap in individual checkpoint files. After the exploration, the agents send the checkpoints of their submaps to the server.
2. The server uses the feature vectors of the first keyframes of the submaps for loop detection based on cosine similarity.
3. If a loop is detected, the PGO is done based on the registration of the pointclouds between the submaps.

If I am wrong, @VladimirYugay please correct me.

[ShilangChen1011]

Thanks for your explanation.

But I'm still confused. Why is it necessary to create so many submaps? Wouldn't it be sufficient for an agent to create just one submap?

Description In Section 3.3 you note that

“We found that directly registering Gaussian means from different agents is unreliable, as Gaussians representing overlapping regions can have widely varying distributions across agents. To address this, we anchor an input point cloud at the start of each sub-map and use it for registration.”

However, if two sub-maps’ starting point clouds have no shared field of view, it seems there would be no overlap to drive a reliable alignment.
Questions

1. Does your method implicitly assume that all agents begin mapping from the same physical location (thus guaranteeing some overlap in their initial point clouds)?

2. If not, what mechanism ensures sufficient correspondence between non‐overlapping start clouds so that registration can succeed?

Thanks for any clarification!

Hi, though I am not the author, I have been playing on this project for a few weeks. So let me try to clarify a bit:

1. Unlike some other multi-agent SLAM works in which each agent only generates a single submap, in MAGiC-SLAM, each agent generates multiple submaps and saves (1)some meta information (including the feature vectors of the first keyframe of the submap), (2)the gaussian model parameters and (3)the point cloud of the submap in individual checkpoint files. After the exploration, the agents send the checkpoints of their submaps to the server.
2. The server uses the feature vectors of the first keyframes of the submaps for loop detection based on cosine similarity.
3. If a loop is detected, the PGO is done based on the registration of the pointclouds between the submaps.

If I am wrong, @VladimirYugay please correct me.

@lemonci @VladimirYugay

[lemonci]

Thanks for your explanation.

But I'm still confused. Why is it necessary to create so many submaps? Wouldn't it be sufficient for an agent to create just one submap?

There are several reasons to create multiple submaps:

1. In real world deployment, if an agent fails to transmit the single submap file to to server because of network bandwidth/connection issues, all data from that agent will be lost. On the other hand, if the agent transmit multiple submap files to the server, probably some of them will be delivered to the server even some of them fail.
2. As the agent expands the exploration area, the submap will grow larger and larger. Even if we use a mask to prevent 3D Gaussian parameters updating in areas already reconstructed well (like in SplaTAM), they are still kept in the GPU so it is not very VRAM efficient.
3. If each agent only maintains one single submap, it has to store all keyframe feature vectors for loop detection. If you also keep the current registration practice in MAGiC-SLAM, aka, directly callingo3d.pipelines.registration.registration_icp like in

   MAGiC-SLAM/src/utils/magic_slam_utils.py

   Line 168 in 4ac5aa9

   fine_alignment = o3d.pipelines.registration.registration_icp(

   , you have to also transmit the point cloud generated from each keyframe’s depth image (see remark), or at least the depth image themselves, which is huge to transmit from agent to server.
   On the other hand, if you choose to use the whole submap’s entire point cloud to do registration, this method will probably fail, because the point clouds from different agents are only similar near the two keyframes where the loop is detected and vary a lot in other parts. If you need to “cut out” the part of point cloud relevant for registration, it is equal to creating multiple submaps. On the other hand, if you insist to keep one agent having a single submap, you have to find another way to do registration.

Remark:
More specifically, the agents in MAGiC-SLAM do not directly transmit the point cloud themselves but the checkpoint files containing the point clouds from agents to the server:

https://github.com/VladimirYugay/MAGiC-SLAM/blob/4ac5aa92ab70139546b42927b62e4f2fd6940222/src/entities/agent.py#L213C1-L216C48

Actually this is not very space efficient because the tensor of the point cloud is about two time larger in file size compared to the original ply files, also we need to rely on np2ptcloud and ptcloud2np to convert them. However, unless you write your own serialization/deserialization in C++, otherwise at this point we cannot avoid using tensor because open3d pointcloud files are not pickleable, so it cannot be directly sent via pipe.send(): isl-org/Open3D#218

[ShilangChen1011]

Thanks for your detailed explanation.

I have another question:

Why do centralized systems like CCM-SLAM typically create a single submap per agent and only upload a portion of keyframes to the server?

They manage data through submap-keyframe-map point relationships.

In contrast, MAGiC-SLAM requires uploading entire submaps and managing them locally.

Similarly, methods like MonoGS and SplaTAM also only create one submap locally.

What are the key factors that make MAGiC-SLAM's approach different in terms of submap management and data upload?

@lemonci @VladimirYugay

[VladimirYugay]

Hey there! Thanks for your interest in our work!

@lemonci is entirely right about your first question.

Another reason for sub-maps is scalability. If the sequence is long enough, any of the above methods will exhaust video memory.

Another reason is speed. If you have a whole map, then doing tracking the way we do will tremendously slow down as the scene grows. We tried to use the MonoGS tracking module instead of ours, but the tracking accuracy was lower.

[lemonci]

Thanks for your detailed explanation.

I have another question:

Why do centralized systems like CCM-SLAM typically create a single submap per agent and only upload a portion of keyframes to the server?

They manage data through submap-keyframe-map point relationships.

In contrast, MAGiC-SLAM requires uploading entire submaps and managing them locally.

Similarly, methods like MonoGS and SplaTAM also only create one submap locally.

What are the key factors that make MAGiC-SLAM's approach different in terms of submap management and data upload?

I am not very familiar with CCM-SLAM (I may ask my supervisor about it but it would be next Friday) but one critical point is that the data structure. The local map CCM-SLAM reconstructed on the agent side is point cloud which doesn’t need to be intensively optimised on GPU. Also, CCM-SLAM only maintains most recent N keyframes and delete the previous keyframes, while MAGiC-SLAM(and other GS-based slam) does not contain such mechanism. So specifically speaking, the mapping on the agents in CCM-SLAM is not complete on the agent side and the agents do send all of the keyframes to server for the complete mapping and loop closure if the network bandwidth allows.

MonoGS and SplaTAM are single-agent GS-SLAM so they don’t need to send anything. Also they don’t have loop closures so they don’t necessarily need to keep the keyframes.

[lemonci]

Hey there! Thanks for your interest in our work!

@lemonci is entirely right about your first question.

Another reason for sub-maps is scalability. If the sequence is long enough, any of the above methods will exhaust video memory.

Another reason is speed. If you have a whole map, then doing tracking the way we do will tremendously slow down as the scene grows. We tried to use the MonoGS tracking module instead of ours, but the tracking accuracy was lower.

Actually I also have a related question for @VladimirYugay regarding the point cloud: why did you choose to transfer the point cloud directly instead of transfer the rgbd image of the first frame from the agent to the server? I think the size of point cloud tensor is larger than the size of rgbd image of the first frame of the submap.

[VladimirYugay]

Yes, sending the RGB-D image would be way cheaper in size, but the computation time on the central server will be larger. We decided to optimize for speed on the server, rather than the size, but both options are viable.