Feature Request: Batch Processing Support for LLM API Integration #3752
Description
Feature Request: Batch Processing Support for LLM API Integration with Medhelm
Overview
We would like to request the addition of robust batch processing capabilities to your Large Language Model (LLM) API, specifically to facilitate seamless integration with Medhelm. The goal is to enable efficient, scalable processing of dataset inputs in batches, evaluation of responses, and structured result storage, while supporting optional intermediate measurements for tracking progress. This enhancement should be designed to work without requiring any modifications to the Medhelm repository.
User Story
As a developer integrating your LLM API with the Medhelm platform, I want to process dataset inputs in batches,
evaluate LLM responses using predefined metrics,
and store the results in a structured format,
so that I can efficiently monitor and benchmark LLM performance
without making any changes to the Medhelm repository.
Acceptance Criteria
- The API supports processing of dataset inputs in batches for LLM tasks.
- Batch and final results are written with relevant metadata (e.g., batch ID, timestamp).
- The system gracefully handles errors, logging issues without halting the overall process.
- Key configuration parameters, including batch size and evaluation metrics
Thank you for considering this feature request. We believe this enhancement will streamline LLM evaluation workflows and improve large-scale task management within Medhelm.