This repository includes use case folders with consensus-defined Testing and Evaluation Frameworks. When completing the CHAI Applied Model Card, review use case specific Testing and Evaluation Frameworks for additional CHAI consensus-defined guidance.
A Testing and Evaluation (T&E) Framework includes a consensus-defined set of methods, metrics, and benchmarks for developers and implementers to more concretely evaluate the responsible use of health AI solutions for a given use case. Content within a T&E Framework varies based on use case and type.
By displaying this content on GitHub we:
- Welcome feedback and new evidence/examples for or against the use of a particular method or metric for a given use case. We have two methods for individuals to provide feedback once information is posted to the public repository.
- For those who use GitHub, we have created a GitHub Issue form.
- For those who would rather not use GitHub, we have created a Microsoft Form, which will automatically generate a GitHub Issue in the repository.
- Issues raised will be prioritized, changes determined, committed, and reflected through version control (see governance.rst and contributing.rst).
- Aim to stay relevant and up to date in a rapidly changing and growing field. We recognize that methods, metrics, and evidence in some areas are emerging, while in others they may be more established.
- Want to prioritize getting information to our community rapidly and in a way that is useful. Waiting for documents to be “ideal” in a space where there are still uncertainties and unknowns, often delays dissemination.
- Want to support the development of an open-source community that includes CHAI members and other domain experts who can begin testing these methods and metrics, providing examples, and filling any gaps in information.
- Important Notes:
- The content you see below will have blanks. This may be because it was outside of the scope of a specific workgroup focus, because the domain was added after content development, or because information still needs to be added. We welcome these additions from our community and other domain experts.
- Content that is included was agreed upon by workgroup members and leads (see governance.rst and contributing.rst).
Please read contributing.rst and governance.rst to learn more about our contributing guidelines and standard operating procedures for open-source projects.
This repository contains living documents that are continuously evolving. The content herein is a work in progress, subject to change, and may not always reflect the latest best practices, guidelines, or recommendations. Users are encouraged to review and use the content at their own discretion. Contributions, feedback, and discussions are always welcome to help refine and improve the material over time.
Copyright (c) 2025 Coalition for Health AI, Inc.
The contents of this repository are licensed under the CC BY 4.0 license.