Add Annotation Data to Test Sets from Spans

[mmabrouk]

Current Behavior

Users can create test cases from spans and map span data to test set columns.

Requested Feature

Add the ability to include annotation data when creating test sets from spans.

Use Cases

1. Expected Answers: Annotate spans with expected answers, then create test sets with LLM inputs + annotated expected outputs
2. Test Case Metadata: Tag spans as "edge cases" or other categories, then add these tags as metadata/columns in the test set for filtering and organization

User Workflow

From a user request:

I'd also like to share what workflow would work best for us regarding annotations, and I'm curious to hear your thoughts:

1. See a trace (either produced from the playground, an evaluation, or a call made in production)
2. Annotate it
3. Append it to a test set, with the annotation having its own column (creating one on the spot if none exists in the test set)

This would serve as the ground truth, increasing the LLM judge's reliability. For example, we are testing how a chat model performs in a specific scenario (mid-conversation). We generate a response and then feed that, along with the annotation, to a judge. The judge has its own evaluation rubric but may still produce false positives or negatives. Regardless of the judge's output, the annotation of what we expect to happen in a specific scenario should help align its reasoning with the expected outcome. In other words, we are giving it a bias towards what its reasoning should be.

[mmabrouk]

Technical Implementation Notes

At a high level, this is straightforward to implement. The main challenge is in our data model: annotations are currently indexed as a list of spans rather than a dictionary. This means:

• No guaranteed order for annotations
• Possible multiple annotations with the same key

Required changes:

• Restructure annotations from list to dictionary in the frontend
• Update the mapping logic to use JSONPath for selecting the first element of each annotation key

What do you think @ardaerzin ?

[ardaerzin]

Restructure annotations from list to dictionary in the frontend

yes this makes sense combined with the mapping logic. need to spend some time laying out the mapping, ie, the first element or a more sophisticated approach, but overall this is the way to go