How can I split/reduce the SQUAD dataset?

[Nomiluks]

I am loading the squad_v2 dataset. But I want to use a smaller number of examples in training and testing. I wasn't able to find any help regarding this.

Code is written below:

from datasets import load_dataset, load_metric

squad_v2 = True
datasets = load_dataset("squad_v2" if squad_v2 else "squad")
print(datasets["train"].shape, datasets["validation"].shape)
# (130319, 5) (11873, 5)

Is there any possibility to reduce the data size from 130319 to any number x that I want?

Thank you

[KaijuML]

Hello,

You can use, the following code:

from datasets import load_dataset

train_size = 100
valid_size = 10

split = [
    f"train[:{train_size}]",
    f"validation[:{valid_size}]"
]

SQuAD_train, SQuAD_valid = load_dataset("squad_v2", split=split)

assert SQuAD_train.shape == (100, 5)
assert SQuAD_valid.shape == (10, 5)

Find more options in the official documentation.

Regards,