Which model weights to consider when you are fine-tuning the second time ?

[teddcp2]

hi all,

I have a doubt on the transfer learning part of the course where Feature extraction has to happen before we do the finetuning as it will be needing the same model weights to optimize further. Now what if I want to fine tune second time/third time? Which model to choose?

Should I take the feature-extracted model and increase the trainable layers from bottom with a lower learning rate ? OR From the last point i.e. I should take the same model where I just did the fine tune and start from the last epoch with a lower learning rate?'

@mrdbourke , Please suggest which would be a correct approach.

[realnihal]

Theoretically, if you use an approach that can dynamically change the learning rate, there is no need for fine-tuning multiple times. But if you fine-tuning on fixed learning rate, then lowering the learning rate on every attempt might yield better results. Also you don't change or choose any different model, you simply recompile it.

[teddcp2]

So, I just need to recompile it every-time after changing the number of unfrozen layers?

[mrdbourke]

Hey there,

Your thoughts here would line up with how I would do it:

Should I take the feature-extracted model and increase the trainable layers from bottom with a lower learning rate ? OR From the last point i.e. I should take the same model where I just did the fine tune and start from the last epoch with a lower learning rate?'

In essence:

• Feature extract
• Unfreeze some layers
• Fine-tune
• Unfreeze
• Fine-tune
• Unfreeze
• Fine-tune

etc...

Though where the best results will come from is a little bit of guess and check.

Jeremy Howard explains this methodology quite well in https://arxiv.org/abs/1801.06146 (see the ablation studies for different fine-tuning methods).

@realnihal's explanation above could also get quite good results, see https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/LearningRateScheduler to dynamically lower the learning rate during training.

[teddcp2]

After unfreezing 5 layers and fine-tuning it, I would increase the unfrozen layers count to 10 and need to recompile.. then finetune it.. correct ? @mrdbourke

With LearningRatescheduler, we can get a proper rate where the model learns the most and pick that for the rest of model training.

[mrdbourke]

After unfreezing 5 layers and fine-tuning it, I would increase the unfrozen layers count to 10 and need to recompile.. then finetune it.. correct ? @mrdbourke

This is correct.