datasets generation

[Kaiseem]

hi,

Well done on applying DPO on generative retrievel!

could you provide the instruction of generate datasets/processed/msmarco-data/train_data_top_300k/pretrain.t5_128_10.pq.300k.json ?

I’d really appreciate your guidance!

[kidist-amde]

Hi! 👋

Thank you for the kind words and for your interest in DDRO !😊

To generate the file datasets/processed/msmarco-data/train_data_top_300k/pretrain.t5_128_10.pq.300k.json:
This file corresponds to the top-300K MS MARCO subset, sampled based on relevance frequency.

You can find detailed instructions on how to:

• Sample the top-300K documents,
• Generate PQ-based and URL docids,
• And create training instances like pretrain.t5_128_10.pq.300k.json

in the following section of the repo:

🔗 src/data/data_prep/README.md

Let me know if anything's unclear, happy to help!

Best,
Kidist 🙌