Hi,

https://github.com/mlcommons/inference/tree/master/language/llama2-70b provides processorca.py to generate the eval dataset, but several other models as https://github.com/mlcommons/inference/tree/master/language/deepseek-r1 and https://github.com/mlcommons/inference/tree/master/language/llama3.1-8b do not provide a reproduction script to generate the evaluation data.

This is an issue if e.g. one would like to apply the evaluation from mlperf on other models that may use other tokenizers.

Could the code used to generate this preprocessed data be shared?

Thank you