Narek Daduryan
UCLA
COM SCI 260C Deep Learning
Learned optimizers are able to outperform traditional optimizers on a variety of machine learning tasks. However, their high computational cost makes it difficult to be practically usable. The ability to transfer a learned optimizer may prove beneficial to offset their high initial cost. We examine the transferability of learned optimizers across different domains. While learned optimizers generally can be transferred to variations in model architecture, they may lose their competitive advantage compared to traditional optimizers. Large changes in models may mark the learned optimizers obsolete when transferred. However, we note that transferring learned optimizers across datasets, or with small architectural variations, may present an advantage in using learned optimizers.