gelectra-base model defination for distillation

[OriAlpha]

Hi all,
I am trying to create a distill version of gelectra-base model, as known we need model defination. I could not find in paper not at least full? So is there any way i could achieve this?? or can i import from transformers as config

[bogdankostic]

Hey @OriAlpha! I'm not sure to what exactly you are referring to with model definition. Could you please elaborate on what information you need? Then I can help you finding that information. :)

[MichelBartels]

Oh, I think I was a bit unclear in my answer. In my sample code, I have already halved "num_hidden_layers" from 12 to 6 and you‘ve probably now halved "num_attention_heads" which causes this issue.
I would also recommend using a larger dataset like the German wikipedia. You can find Wikipedia dumps here (dewiki-latest-pages-articles.xml.bz2).

[OriAlpha]

that exactly correct i reduced num_attention_heads, i was also confused at this point. I have few questions, regarding sample code,

processor = Processor.convert_from_transformers(tokenizer_name_or_path=teacher_name, task_type="question_answering",
                                                max_seq_len=512, doc_stride=128)

1. i believe processor converts transformers to haystack version model right (correct me if i am wrong).
2. If i want to load an external model (like fine-tuned model) do i have have to go through 'processor' step

[MichelBartels]

1. The processor takes care of preprocessing (doc splitting, tokenization etc.)
2. If you have a fine-tuned model uploaded on HuggingFace, you can skip most of the code and just call:
   student = FARMReader("organization_name/model_name")
   If it's saved on your device in HuggingFace format, you just have to execute the following part of the code:

student.save("finetuned_student_adaptive_model")
processor = Processor.convert_from_transformers(tokenizer_name_or_path=teacher_name, task_type="question_answering",
                                                max_seq_len=512, doc_stride=128)
processor.save("untrained_student_adaptive_model")
del student, processor
student = FARMReader("untrained_student_adaptive_model")
teacher = FARMReader(teacher_name)

[OriAlpha]

i trying to use a finetuned model, which is for ElectraForSequenceClassification. When i change config (e.g. change no of class 3 ). i am getting error in converting model
student = Converter.convert_from_transformers("untrained_student", "cpu")
error as follows:

size mismatch for weight: copying a param with shape torch.Size([3, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).
size mismatch for bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).

I have a pretrained model for SequenceClassification hence i changed to

student_config = ElectraConfig(**student_config)
student = ElectraForSequenceClassification(student_config)

[MichelBartels]

Hi @OriAlpha, intermediate layer distillation is currently only implemented for QA (2 classes). That should not be a problem, however, because intermediate layer distillation is not performed over the prediction heads. Because of that, I would recommend keeping the number of classes at 2. Changing it later on will not make a difference as the weights of the prediction head will not be optimized during intermediate layer distillation.

[OriAlpha]

I have tried whole process, but distilled model results in accurcay drop (i.e., around 10%), i tried for training for more epochs and dont see any improvements. any suggestion to improve distilled model performance.

[MichelBartels]

Hi @OriAlpha, I'm sorry for answering so late. How did you fine-tune it for the downstream task? If not, have you adapted the task-specific distillation and the data augmentation? How big are your datasets for general distillation and task-specific distillation? For how many epochs did the distillation run?