Skip to content

Allow A, B, C, D enumeration of question options #1853

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

johnjosephhorton
Copy link
Contributor

Summary

  • Add support for letter-based enumeration (A, B, C, D) of multiple choice question options, providing better compatibility with existing evaluation frameworks
  • Implemented through a new enumeration_style parameter with options "numeric" (default) and "letter"

Test plan

  • Added unit tests for the new functionality
  • Test that letter enumeration works with HTML rendering
  • Test that letter codes (A, B, C) can be converted to their corresponding options
  • Test case insensitivity for letter codes
  • Test proper error handling for invalid letter codes

Fixes #1222

🤖 Generated with Claude Code

johnjosephhorton and others added 8 commits April 5, 2025 18:47
…ter()

This change fixes issue #1715 by:
1. Adding support for multi-line strings in the Results.filter() method
2. Allowing template-style syntax with {{ }} notation in filter expressions
3. Adding comprehensive tests for both features

The fix normalizes whitespace and newlines in filter expressions, making
filter() more robust and user-friendly when dealing with complex filter conditions.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Add enumeration_style parameter with "numeric" and "letter" options
- Update templates to support A, B, C style option labeling
- Handle letter responses in validation and answer processing
- Add tests for the new functionality

Fixes #1222 - Allow A, B, C, D enumerations of question options

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@zer0dss
Copy link
Collaborator

zer0dss commented Apr 7, 2025

@johnjosephhorton I'm working on fixing the failed coopr test. I think there is an extra word entered in the prompt.

@zer0dss
Copy link
Collaborator

zer0dss commented Apr 7, 2025

Actually there is no word change. I'm checking if there are price changes on coopr project.

@zer0dss
Copy link
Collaborator

zer0dss commented Apr 7, 2025

The price changes due to an edsl change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Consider allowing A, B, C, D, etc. enumerations of questions options to be consistent with some evals
3 participants