SeTok

Towards Semantic Equivalence of Tokenization in Multimodal LLM

This is the repository that contains the source code for the SeTok project page.

If you use SeTok in your project, please kindly cite:

@article{wu2024towards,
  title={Towards Semantic Equivalence of Tokenization in Multimodal LLM},
  author={Wu, Shengqiong and Fei, Hao and Li, Xiangtai and Ji, Jiayi and Zhang, Hanwang and Chua, Tat-Seng and Yan, Shuicheng},
  journal={arXiv preprint arXiv:2406.05127},
  year={2024}
}

Website License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.