UniCodec (ACL 2025 Main)

UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook
Yidi Jiang,Qian Chen,Shengpeng Ji,Yu Xi,Wen Wang,Chong Zhang,Xianghu Yue,Shiliang Zhang,Haizhou Li
National University of Singapore; Tongyi Speech Lab

In this work, we introduce UniCodec, a unified audio codec with a single codebook to support multi-domain audio data, including speech, music, and sound.

To achieve this, we propose a partitioned domain-adaptive codebook method with domain Mixture-of-Experts strategy to capture the distinct characteristics of each audio domain. Furthermore, to enrich the semantic density of the codec without auxiliary modules, we propose a self-supervised mask prediction modeling approach.

As a single unified codec model, UniCodec achieves superior subjective reconstruction performance while maintaining a high compression rate in all three domains (speech/music/sound).

Installation

conda create -n unicodec python=3.9
conda activate unicodec
pip install -r requirements.txt

Train

python train.py fit --config ./configs/xxx.yaml

Infer

Model checkpoint 🤗 is available in Huggingface.

python infer_audio.py

Citation

@article{jiang2025unicodec,
  title={UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook},
  author={Jiang, Yidi and Chen, Qian and Ji, Shengpeng and Xi, Yu and Wang, Wen and Zhang, Chong and Yue, Xianghu and Zhang, ShiLiang and Li, Haizhou},
  journal={arXiv preprint arXiv:2502.20067},
  year={2025}
}