Support for sequence labelling tasks

[penguinwang96825]

Hi

First of all, thanks for the great work on evalscope, it’s been really useful for evaluating QA and MCQ tasks!

I wanted to ask if there are any chances to support sequence labelling tasks (e.g., NER, POS tagging). This would be particularly valuable for evaluating models on token-level or span-level outputs, beyond just QA/MCQ.

I’m thinking maybe we could add support for sequence labelling via in-text annotated outputs. The approach is to represent labels inline using XML-like tags, for example:

Input: Yang lives in Manchester  
Output: <response> <person> Yang </person> lives in <location> Manchester </location> </response>

The framework then extracts the corresponding BIO or tagging labels from model outputs and evaluates them with standard metrics (e.g., seqeval F1).

Thanks again for maintaining this great library!

Warm regards,
Yang

[Yunnglin]

Thank you for your kind words about evalscope! We're glad it's been helpful for your evaluation tasks.

Your suggestion for supporting sequence labelling tasks (NER, POS tagging) is excellent. The XML-like annotation approach you proposed is a clean and practical solution for handling token-level evaluations.

We'll definitely consider this feature for our development roadmap. In the meantime, we encourage you to submit a PR if you're interested in implementing this functionality - we'd be happy to collaborate and provide support!

[penguinwang96825]

@Yunnglin I’d be happy to implement this. I want to make sure I’m aligned on the design before I start.

High-level plan:

1. Keep the dataset ground-truth in its native BIO format (e.g. CoNLL-2003).
2. Create a pipeline that prompts the LLM to produce an XML-like prediction (wrapped in a top-level <response>...</response>. Entity spans wrapped like Yang or Manchester ).
3. Implement a function that converts the model’s XML-like prediction back into BIO tokens aligned with the dataset tokens.
4. Compute token/span metrics with seqeval (e.g., overall F1) comparing ground-truth BIO to the converted BIO predictions.

Design decisions where I’d appreciate guidance:

1. Where to implement this in the codebase? I'm thinking subclassing DefaultDataAdapter just like QA task under folder evalscope/benchmarks/conll2003.
2. How to handle malformed model output? LLMs often return imperfect XML (e.g., missing closing tags, tag name mismatches, extra text). Is there any existing evalscope support for structured-output or retry mechanism utilities?

If that all sounds reasonable I’ll prepare to draft a PR. Happy to start as soon as I get your guidance on the above.

[Yunnglin]

Thank you for the detailed plan! Your high-level approach sounds very reasonable. Here's the reference documentation for adding evaluation benchmarks: https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/add_benchmark.html. I'd recommend reading through it first.

Regarding your design decisions:

Implementation location: Yes, subclassing DefaultDataAdapter and creating a new folder evalscope/benchmarks/conll2003 is the right approach. You can reference the general_qa implementation as a template.

Handling malformed model output: Currently, there's no built-in fallback or retry mechanism for incorrectly formatted responses. However, you can set the repeats parameter, which will evaluate each sample n times repeatedly, allowing you to calculate metrics with better statistical support (similar to how HumanEval handles this).

Your overall plan looks solid. Feel free to proceed with drafting the PR, and we can iterate on the implementation details during the review process. Looking forward to seeing your contribution!