Overfitting

[gauravreddy08]

Hey Daniel,

Recently I figured that it is not about what complex model you use but its how you use it.

Suppose you get a val_acc 0.77 on EfficientNetB0, So if you try to use EfficientNetB4 you wont probably get better results cause your model overfits on the data.

Overfitting is constant pain in a** when we use transfer learning. So,

• How one can solve it?
• I've heard of Dropout Layers, What are they?
• Does Data Augmentation always solve overfitting, cause in my case, when I augment the data the model Underfits?

[mrdbourke]

Hey Gaurav,

There are several different ways to approach overfitting (image taken from: 03 Convolutional Neural Networks Slides):

I've heard of Dropout Layers, what are they?

Dropout layers randomly remove a certain number of connections between two layers (e.g. layer has 32 neurons, with a dropout rate of 50%, only 16 of them will connect to the next layer).

Removing a certain number of connections has the hope of improving the remaining connections.

See this explanation for more: https://stats.stackexchange.com/questions/241645/how-to-explain-dropout-regularization-in-simple-terms

There's a dropout layer available in TensorFlow too: https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout

Does Data Augmentation always solve overfitting, cause in my case, when I augment the data the model Underfits?

Not always. I'd try reducing data augmentation/trying different kinds of augmentation.

I'd look into methods such as RangAugment and AutoAugment (RandAugment seems to perform better in practice).

Other overfitting methods you might want to research include (often referred to as regularization):

• Weight decay (decrease weights over time)
• Learning rate decay (decrease learning rate over time, though is often referred to as a different training method)
• Early stopping (stop model training before it trains too long)
• Label smoothing