Fine-tune for non-English languages

[fatjoni]

I am looking on finetuning opt or opt-iml-max for my native language, Albanian. I need to make sure that i am on the right path for a generative question answer model. My idea is that i can finetune opt-iml-max with an extra Albanian corpus of books and artickles (ideally so that the model is more aware of the language) and then fine tune it on a question answer context corpus. Am i correct?

[binmakeswell]

Hi @fatjoni Yes, you can finetune a pretrained model with your data through Colossal-AI. Although the training language is different, the language ability of the models is similar.