[실험 결과] Optimizer 3종(Adam, AdamW, MADGRAD) 비교 SATRN_mini

[remaindere]

실험 조건 : CyclicLR Scheduler, CE Loss, augmentation 상하좌우 flip p=0.5

요약 : MADGRAD > AdamW > Adam

lr = 5e-4
weight_decay = 1e-4
MADGRAD : (params, lr=lr, momentum=0.9, weight_decay=weight_decay, eps=1e-06)
AdamW : (params, lr=lr, betas=(0.9, 0.999), eps=1e-08, weight_decay=weight_decay, amsgrad=False)
Adam : optim.Adam(params, lr=lr)