This project further pre-trained the Mistral model and fine-tuned it with instructions using LoRA, based on five welding reference textbooks, aiming to develop a large language model specialized in the welding domain.
- model:
unsloth/Mistral-Nemo-Base-2407 - Parameter-Efficient Fine-Tuning : LoRA
- Hardware: Colab A100, 40G
For data preparation,
pip install langchainModel training will run on Colab so you don't need to install dependencies locally.
The preprocessed data is here, which contains training and evaluation data.
If you want to use the shared evaluation data, which only contains multiple-choice questions, please see eval/README.md before using it.
If you want to prepare your own training data, please see preprocess/README.md for more details. The data will be generated in data folder and the data_example folder is provided for your reference.
See finetune/README.md for more details.
This project uses a dataset with multiple-choice questions to evaluate the model as mentioned in Data Preparation section.
The initial result is here.
- Higher-Quality training and evaluation data
- Larger LLM / Train more parameters