Add AI2ARC dataset

[TomeHirata]

Add a dataset for https://huggingface.co/datasets/allenai/ai2_arc

[Copilot]

Pull Request Overview

This PR adds support for the AI2 Reasoning Challenge (ARC) dataset via a new AI2ARC class and related utilities.

• Introduces AI2ARC to load and process the ARC-Challenge and ARC-Easy subsets into DSPy examples.
• Adds ai2_arc_metric and parse_arc_answer functions for evaluation and parsing model outputs.
• Exposes AI2ARC in dspy.datasets and provides convenience loaders ARC_Challenge and ARC_Easy.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
dspy/datasets/ai2_arc.py	New dataset loader, processing logic, metric/parsing utilities
dspy/datasets/init.py	Registers AI2ARC in the module’s public API

Comments suppressed due to low confidence (2)

dspy/datasets/ai2_arc.py:144

• [nitpick] Function names should follow snake_case per PEP8. Rename ARC_Challenge to arc_challenge (and similarly ARC_Easy).

def ARC_Challenge():

dspy/datasets/ai2_arc.py:84

• Consider adding unit tests for ai2_arc_metric and parse_arc_answer to verify that answer parsing and evaluation behave as expected.

def ai2_arc_metric(gold, pred, trace=None):

[chenmoneygithub]

This is pretty interesting, I am down to checking in this one.

I will defer the decision to @okhat if we want to expand the dataset collection.

[arnavsinghvi11]

to my knowledge, the datasets we have in the repo only exist to maintain some past example notebooks/tests (though maybe we should actually just remove them given that we've ported these into tutorials - e.g. I don't see any instances of dspy.datasets.colors).

it's probably just cleaner to refactor out these datasets + their references in tutorials (so just adding ai2arc for demonstrative purposes in a tutorial) and only keep Dataset and DataLoader in dspy/datasets/

[okhat]

yeah it’s rare that we add datasets anymore 😅

[TomeHirata]

Oh, thank you for the context. So we won't add more datasets and will deprecate existing datasets someday?