There are too many cycles

[csm-coder]

Dense-Scene-Matching/libs/engine/launcher.py

Line 27 in 4957fa3

for i in range(self.cfg.TRAIN.train_iters):

should the "self.cfg.TRAIN.train_iters " be changed to "int(self.cfg.TRAIN.iters/self.cfg.TRAIN.model_save_iters)"? otherwise，there are too many cycles.we will save 1600000 models?

[Tangshitao]

Yes, you can use fewer iterations. The one provided in configuration is not optimal.

[Tangshitao]

The released training codes are not guaranteed to reproduce the results in paper, but you can get very similar results.

[csm-coder]

i have trained my model basing on the best_model you provided,and also try to train a model without the best_model.but both of them can't get correct results。the errors of result are too large。the following are some loss curve and the test results in 7scenes/heads.


the model trained without best_model,batchsize=2


the model trained without best_model,batchsize=4


the model trained with best_model,batchsize=8,freeze_backbone=true

i have only changed the iterations, the batchsize and whether load best_model or not. but the results are obviously incorrect。
I am so confused about that。

[Tangshitao]

Do you train the model on 7scenes dataset? I don't know the training curve when training the model on 7scenes dataset, but the initial loss seems to be too large when you use model_best.pth as pretrained model. It seems the model is not load correctly. Can you provide the initial training loss?

[csm-coder]

no,i train it on scannet dataset and test on 7scenes/heads. i am also confused why the initial training loss is so large. when i load the best_model.pth to test, it give correct result similar to the results in paper.this is the train loss curve when i use model_best.pth as pretrained model. http://115.156.215.223:8891/

[Tangshitao]

I think the model does not load correctly in the training model. Can you check all parameters are loaded when training?

[csm-coder]

i have checked. the model.state_dict is loaded successfully in the training model which is the same as testing. testing and training load the best_model in the same way. the testing result is correct, so i think the model loading should have no problem。Could there be a problem elsewhere?

[Tangshitao]

Then the problem is with data. Can you check whether the data is load correctly? For example, the unit of depth should be meters when fed into the model.

[csm-coder]

i try to train on 7scenes dataset with best_model.pth. the loss curve Looks better than it on scannet dataset. and the trained model give a correct result.


so I consider whether there is a problem with my scannet dataset.my scannet dataset is scannet_v2,and its file path is a little bit different from your scannet label file,i have changed it. is your scannet dataset scannet_v1? and does it matter?

[Tangshitao]

I also use scannet_v2, but I also think there must be something wrong with dataloader. I cannot upload the one I use since it is too large and copyright issue. Please check the depth reading codes carefully.