Data processing

[Jayant1234]

Which script is creating the processed data in pkl format?
as used in data_utils.py file:
CODEX_PREDICTIONS_PATH = 'processed_data/codex_predictions.pkl'

I am trying to use your code as a baseline, please help.

[zhouzihao501]

I want a processed data too. Thanks

[FranxYao]

Hi all,

The processed data are now uploaded in https://drive.google.com/drive/folders/1BOXcUTnEyvQia_ypHcaUnUbLsN4HzqmQ?usp=sharing

To see how the processed data look like, use note book:
https://github.com/FranxYao/FlanT5-CoT-Specialization/blob/main/notebooks/inspect_processed_data.ipynb

Cheers!

[Jayant1234]

Hi @FranxYao , How do you mix the in_context_chain_of_thought.pkl? It seems to contain 27x or so repetition of the whole training dataset. I want to know because changes I make in that 200000 entry pkl file are time-consuming and I would rather make it on 8000 entry file and just repeat like you did.