MedPajama: A Large-scale Trustworthy Medical Corpus

MedPajama is an ambitious initiative to build one of the largest, most reliable, and clinically credible medical corpora to support the development of next-generation medical AI systems.

Logo for MedPajama designed with ChatGPT-4o

⚡ Introduction

This project focuses on constructing a trustworthy medical dataset by:

1. Collecting and filtering massive-scale pretraining data from authoritative sources.
2. Annotating data with multi-level labels (e.g., medical domain, semantic type, source credibility).
3. Extracting the highest quality and most reliable data to build a RAG knowledge base.
4. Selecting high-confidence and challenging questions to construct datasets for SFT and RL.

Whether you're developing medical foundation models, disease-specific LLMs, or medical RAG systems, MedPajama provides a trusted data backbone to support your research and applications.

🎯 ToDo

• Collection of Pretraining Data and Domain-Specific Medical Filtering.
• Filter multi-dimensional labeled data and verify it with medical experts.
• Analyze how different levels of data trustworthiness impact medical LLM performance.
• Extract a subset of the highest-trust data to build a RAG knowledge base for retrieval.
• Select a subset of knowledge-rich or challenging data to construct SFT datasets.
• Select a subset of high-difficulty samples to construct RL training data.