Format of training data

[Avs-safety]

Hello,

What format should the data in the data_dir be in the tutorial below? I assume batches of .txt files and are there any special requirements for those .txt files?

Apologies if I have missed it somewhere - Thanks!

reader = FARMReader(model_name_or_path="distilbert-base-uncased-distilled-squad", use_gpu=True)
data_dir = "data/squad20"
# data_dir = "PATH/TO_YOUR/TRAIN_DATA"
reader.train(data_dir=data_dir, train_filename="dev-v2.0.json", use_gpu=True, n_epochs=1, save_dir="my_model")

[bogdankostic]

Hi @Avs-safety! The data_dir is the directory containing the file specified as train_filename. The train file should be in SQuAD format. You can find more details about the dataset here.

[Avs-safety]

Ah ok, thanks @bogdankostic ! and that is SQuAD format rather than the .json that can be exported from the annotation tool? Thank you

[bogdankostic]

The json-file you extract from the annotation tool follows the format of the SQuAD dataset.