what does the training loss curve look like

I'm trying to train SSN via train_ssn.py, but after running ~40,000 iterations there seems to be a lot of jittering but no meaningful decrease in the training loss. I know from reading previous issues that convergence takes ~500,000K iterations, but with my computing resources it would take a few days to reach convergence.

So I was wondering whether the authors could kindly tell me / show me what the training loss curve looks like as a function of iteration number, starting from iteration 0 all the way to convergence.

Thank you in advance.

[varunjampani]

Sorry that I don't have any saved logs or plots for this. I might not find time soon to re-train to produce these plots.

Would it be ok if I instead ask the following two questions?

1. is it not unusual that the training loss sometimes increases during the early iterations? Is it possible for the training loss to decrease after it increases early on?

2. how many iterations does it takes for the training loss to start decreasing in an meaningful way?

[varunjampani]

If I remember correctly, Adam optimization results in a training curve with ups and downs. I do not remember how much ups and downs the loss is fluctuating.

[CYang0515]

@coarsesand In my experiments, the position loss will increase along with iterations, and the reconstruction loss will decrease gradually.

@CYang0515 does decrease in reconstruction loss eventually overpower the increase in position loss, resulting in a decrease in the overall combined loss with increasing number of iterations?