-
|
In the FIM task, codellama should generate only the missing part (i.e., the middle part). However, I noticed that in the data processing, labels are set to input_ids. Could you clarify why this is the case? Specifically, will the fine-tuned CodeLlama model output the full code with the filled-in parts, or will it only generate the missing part (i.e., the middle section)? Thank you for your help! |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
I noticed that the model’s output itself is the complete code with the filled-in parts. This makes sense now, as the model isn't just generating the missing middle part but rather completing the entire code, which aligns with setting labels as input_ids. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Gorgerbin, thanks for your question and the answer, your understanding is correct. This auto-regressive FIM approach was described by Bavarian et. al. in "Efficient Training of Language Models to |
Beta Was this translation helpful? Give feedback.
I noticed that the model’s output itself is the complete code with the filled-in parts. This makes sense now, as the model isn't just generating the missing middle part but rather completing the entire code, which aligns with setting labels as input_ids.