ModuleNotFoundError and model not recognized

[burakverdi-hue]

Hello HELM team,

I’m experiencing two persistent issues when trying to run HELM evaluations, both in Google Colab and in a clean local environment (PyCharm, Python 3.10), despite installing the library directly from source using:

git clone https://github.com/stanford-crfm/helm.git
cd helm
pip install -e .[all]

1. ModuleNotFoundError: No module named 'helm.benchmark.simple_scenarios'
   When attempting to import:

from helm.benchmark.simple_scenarios import MultipleChoiceScenario

I receive:

ModuleNotFoundError: No module named 'helm.benchmark.simple_scenarios'

This happens even after verifying that other modules (like helm.benchmark.metrics.basic_metrics) are present. The simple_scenarios.py module seems to be missing entirely from the installed package.

2. Model not recognized by helm-run

In a separate attempt to run a scenario with:

--models-to-run stanfordhealthcare/llama-4-maverick-17b-128e-instruct-fp8

I encounter the following:

Exception: Unknown model 'stanfordhealthcare/llama-4-maverick-17b-128e-instruct-fp8'

The model is correctly defined in the YAML configuration file, but helm-run fails to recognize it.

Questions:
• Has simple_scenarios.py been removed or renamed in recent versions?
• What is the correct way to define and evaluate multiple-choice tasks now?
• How can we ensure that public models are correctly recognized by helm-run?

Any guidance would be greatly appreciated.

Thanks in advance for your help!

[yifanmai]

Hi @burakverdi-hue,

In the latest PyPI release v0.5.7 and the current main branch, the simple_scenarios module is defined in this file. The full module path is helm.benchmark.scenarios.simple_scenarios. In addition, there is no MultipleChoiceScenario in this file - you are probably looking for SimpleMCQAScenario. Putting these together, the correct command should be:

from helm.benchmark.scenarios.simple_scenarios import SimpleMCQAScenario

The arguments to model-to-run should be the values in the model_name field in model_deployments.yaml, not the name field. For instance, you should use --models-to-run meta/llama-4-maverick-17b-128e-instruct-fp8 instead of --models-to-run together/llama-4-maverick-17b-128e-instruct-fp8

Note that by default, HELM will use Together AI for Llama 4, so you need to have a Together AI API key. If you do not have one, you can run the Hugging Face model locally by following these instructions.

Also note that the model deployments that start with stanfordhealthcare/llama-4-maverick-17b-128e-instruct-fp8 are for Stanford Healthcare internal use only, and does not do anything useful for external users.