Hyperparameter suggestions

[KaiserWhoLearns]

Hi! I am interested in trying different hyperparameters in fine-tuning for T5-base. After checking the doc, I found a gin file with many parameters.

I was wondering where may I find a list of hyperparameters (and some good initial values) to tune on? Currently, I have the below list based on the gin file.

Max sequence length: 512
Learning rate: constant_learning_rate=0.001
Weight decay: 0.0 (No weight decay for original T5)
AdafactorOptimizer.epsilon1 = 1e-30
AdafactorOptimizer.epsilon2 = 0.001
Warmup steps: 10000
Number of epochs
Gradient accumulation steps
Gradient norm: 1.0

Could you please give me some suggestions on hyperparameter suggestions?

[craffel]

I have only ever observed marginal gains from using anything other than the default suggested fine-tuning hyperparameter settings (constant LR = 0.001, dropout = 0.1, etc).

[KaiserWhoLearns]

Thank you for answering!
->default suggested fine-tuning hyperparameter settings (constant LR = 0.001, dropout = 0.1, etc) Would you mind to point me to the default settings, please? Is it in constant_0_001.gin (from which I only found the LR, but seems like there are some other hyperparameters)?

[craffel]

Supplying the LR gin and making sure the dropout is 0.1 should be everything you need.