Only Head is trainable

[Atmoboran]

I was wondering, with what weights the decoder and the head start if both the encoder and decoder are frozen?
I'm getting suprisingly good results in a cloud segmentation task for a model which only has the head weights trainable.
To be precise, the accuracy is just approx 9% compared to the "best" setup with all weights trainable and the pre-trained backbone.

Best regards,
Atmoboran

[Atmoboran]

So I reproduced this with the burn scars example from: https://github.com/IBM/terratorch/blob/main/examples/tutorials/PrithviEOv2/prithvi_v2_eo_300_tl_unet_burnscars.ipynb

So those are the values and plots for:
backbone_pretrained = True
freeze_backbone = True
freeze_decoder = True

And those are the metrics and plots for:
backbone_pretrained = False
freeze_backbone = True
freeze_decoder = True

I think, if this is correct, it is pretty fascinating how well the model performs with random frozen encoder, random frozen decoder and solely a trained head (which I guess is itself a substantial model with 5M parameters).
@Joao-L-S-Almeida proposed the 'Extreme Learning Machine' (https://en.wikipedia.org/wiki/Extreme_learning_machine) as an explanation.

[Joao-L-S-Almeida]

Yes, @Atmoboran . I suggested ELM as an analogy, since we have a large network with many fixed and randomly initialized parameters where just the last layer is trainable. Maybe the Universal Approximation Theorem is applicable here, but I would like to have a more formal opinion about it.

[Joao-L-S-Almeida]

Hi, @Atmoboran . Maybe you could be interested in testing an experimental branch for using Mixture of Experts (MoE) for the head netowork. You can check it here.