Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval

🔥 News

• [2025/06/05] We update our inference code, both on github and huggingface.
• [2025/06/01] We release our training code and inference code.
• [2025/05/28] We release our models and training datasets to HuggingFace.
• [2025/05/26] We release our paper: Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval

🛰️ Release Plan

• Paper
• Models, Datasets
• Inference code
• Training code
• Evaluation code

👀 Overall

Universal multimodal embedder, UNITE, allows for a unified representation of arbitrary multimodal contents.

🔭 UNITE

Overview of UNITE,: (a) Model architecture utilizing LMM as the backbone, supporting multimodal inputs (text, images, videos, and their combinations). (b) Similarity matrix after applying MAMCL, which enables focused contrastive learning by restricting comparisons to samples sharing the same target modality, thus reducing inter-modal interference.

🏆 Performance

Performance comparison on fine-grained video-text benchmark (CaReBench) and image-text benchmarks (ShareGPT4V, Urban1K, DOCCI).

Performance comparison on instruction-based retrieval benchmarks (left: MMEB and right: WebVid-CoVR).

📷 Visualization Results

🚀 Quick Start

Installation

conda create -n unite python=3.10 -y
conda activate unite

pip install torch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0
pip install flash-attn --no-build-isolation
pip install -r requirements.txt

Data Preparation

• Retrieval Adaptation: Unite-Base-Retrieval-Train
  Please download the raw videos of Tarsier2-Recap-585K from omni-research/Tarsier2-Recap-585, refer to here.

• Instruction Tuning: Unite-Instruct-Retrieval-Train
  Please download the raw images of MMEB-train from TIGER-Lab/MMEB-train, refer to here.

Training

For single-gpu or multi-gpu settings, you can utilize the following training scripts:

bash scripts/multi_gpu/qwen2_vl_2b/stage1_adaptation.sh
bash scripts/multi_gpu/qwen2_vl_2b/stage2_instrcution.sh

For multi-node settings, you can utilize the following training scripts:

bash scripts/multi_node/qwen2_vl_2b/stage1_adaptation.sh
bash scripts/multi_node/qwen2_vl_2b/stage2_instrcution.sh

Inference

You can get the simplest inference code from Huggingface QuickStart, or run the following command.

cd inference_demo
python inference.py

🙇 Acknowledgments

The codebase of UNITE is adapted from LLaVA-NeXT and FastChat. We are also grateful for the following projects our UNITE arise from:

• GME, LamRA, VLM2Vec, E5-V
• Qwen2-VL, InternVideo2
• BEIR, sentence-transformers, img2dataset

📋 Citation

If you find our work helpful, feel free to give us a cite.

@article{kong2025modality,
  title={Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval},
  author={Kong, Fanheng and Zhang, Jingyuan and Liu, Yahui and Zhang, Hongzhi and Feng, Shi and Yang, Xiaocui and Wang, Daling and Tian, Yu and W., Victoria and Zhang, Fuzheng and Zhou, Guorui},
  journal={arXiv preprint arXiv:2505.19650},
  year={2025}
}