Experimental Features

[kllgjc]

• Initialize a bounding box for the splats from the sparse point cloud produced by SfM, similar to NeuS or Neuralangelo? could potentially speed up training by knowing only to produce splats in that region?
• Instead of just crop x,y,z like in sibr viewer, allow for lasso select and delete. Also would be good to be able to rotate the point cloud so XYZ are aligned to the coordinate system you want. I believe this is done with the extrinsics of initial camera pose (like the first in the "input" folder) in colmap.
• View cube for top, front, left, right, iso etc. view (if those are the right direction, a la above^^)
• Ability to add annotations/tags in the viewer, similar to auto desk ReCAP. Can snap to a point on the background point cloud for this?
  - Ideally, if an annotation is placed, it would be able to figure out the "closest camera" poiting at that location, and could include
  that image in a thumbnail.
• Initialize from a LiDAR cloud for more dimensionally accurate reconstruction
• might be a colmap thing, or dataloader to get it in whatever format the train.py script extracts from colmap inputs, but ability to align photos to a point cloud like in agisoft/reality capture
  - Alicevision/meshroom i believe is working on this.
  - Nerftsudio has several dataloaders to get to the transforms.json file, which i believe is usable in this repo, but does not have the initial point cloud, which would make better reconstructions (it think?)

[MrNeRF]

Thanks for tacking the time to provide ideas for feature extension. The backlog is piling up.

Instead of just crop x,y,z like in sibr viewer, allow for lasso select and delete.

What do you exactly mean by lasso select? What should be selected?

[kllgjc]

Something like in Recap. Most point cloud viewers have something like it to cleanup unwanted portions of the cloud. I'm not 100% certain how the gaussian "splats" compare to points in a point cloud, though.

[kllgjc]

Some things I’ve tested out using the original implementation (which you might already know):

When you run training on a dataset the first time it creates a points3d.ply file from the points3d.bin file in the sparse-0 subfolder. This is what it actually uses for the background point cloud. I removed points3d.bin and reran training and it still worked. I then edited the ply file to remove points far away from the region of interest, and it focused more on that portion and less on populating the background with splats, also ran slightly faster!

Then, I tried importing the gaussian ply file produced in the output into cloud compare. It had like 50 scalar fields to import, which i had to do manually, but found this conversation where if you append the text in the ply file with scalar_ it should import it manually. Then you could remove floaters in cloud compare using the segment tool. The henry pierce guy did say they implemented point removal in and experimental SIBR viewer, but in the interim this works for cleaning up the clouds. When you reimport it, you just have save the file as a ply again, and then edit the headers so they match the original (without the cloud compare nonsense and remove scalar_ from the fields).

[MrNeRF]

@kllgjc Thx, nice pointer. Thank you. I was not aware of this. I think it refers more or less directly to creating a bounding box which you can resize and then retrain to focus more on the relevant parts. That might improve the visual quality. I also have looked a bit into how to remove floaters. There are some promising approaches but none of it I tried to implement so far. Currently I invest my spare time into the CUDA transition. Then, if this will be finished, there will come more. Exciting times.

[atreiou]

Hi, I'm new to coding but not programming, I'm pretty good at figuring out how to put together systems, I used to program when I was kid back in the 90s too lol. I am using GP and Claude to help with coding, and I am going to install code llama locally so that will also help. So, not the best and maybe nowhere near good enough to help with anything here, but I am putting together successful programs for other things for myself, so it is possible I could help.

Therefore are there any of the smaller, more straight forward tasks that I could help with? I could lessen the load a little bit, let the pros focus on the big stuff? Also, I think I read in one of your other posts that you "don't support windows?" I haven't actually had a chance to look at any of the project yet but I am windows only, so will this even work on my machine?

[MrNeRF]

Hi @atreiou,
thx for you interest. Any help is welcome. Of course, first a prerequisite is to make this successfully compile and run on your machine.
On windows, you can either follow this guide: #4
Otherwise, you can try to make it run under windows subsystem for linux. Both ways work. However, you have to figure out how it works best for you. Unfortunately, I cannot help you since I don't have a windows machine and no experience what the problems are.

If you manage to do this, you are welcome to ping me again and I will assign you a simple task (also depending on your background). You are welcome to use any help you want (chatgpt, llms in general). I don't mind. However, you should understand what you are doing.

[atreiou]

Ok great, I won't be able to try and get it up and running until sometime this week, but I will keep you posted. If i can get it to work, I'll then practise with it until I understand exactly what's going on, then if I think I can definitely help, I'll let you know. :)