Learning rate compared to adamw

[davnords]

Hi! I see the example code provides 67x increase of learning rate for muon params vs. adamw params. Is this a reasonable heuristic for translating previously used adamw learning rate to now using muon for training?

Thank you,

/ David

[kiriri]

Funnily enough I was just looking into the exact same thing, and I found a learning rate like you would use with adamw to be much more effective than the 0.1 from the example. In my case 1e-3 gave me an accuracy of 0.84 after 1 epoch, while the suggested rate of 1e-1 gave me only 0.36 on the same model/dataset. Even 10 epochs later, the 0.1 lr only reached acc 0.52

[davnords]

@kiriri interesting! Yes I was struggling a little bit on adapting the MAE learning rate to Muon and was using the 67x increase. This seems to indeed be too much for many tasks (in my experience)

[davnords]

Thank you!