Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding

Duo Zheng^*, Shijia Huang^* and Liwei Wang^‡
The Chinese University of Hong Kong

^*Equal contribution. ^‡ Corresponding author.

---

🚀 Announcing Our Latest Work!

We are excited to introduce our newest research: Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors.

This research delves into how explicitly incorporating 3D vision geometry priors can significantly advance the capabilities of Multimodal Large Language Models (MLLMs) in understanding the 3D world through video inputs. We believe this approach marks a significant step forward in creating more spatially aware and robust AI systems.

➡️ Discover more about this exciting new project here: Website Arxiv

This work complements and builds upon our research in creating powerful 3D-aware models. Below, you'll find the details for our related project, Video-3D LLM:

---

Architecture

We propose a novel generalist model, i.e., Video-3D LLM, for 3D scene understanding. By treating 3D scenes as dynamic videos and incorporating 3D position encoding into these representations, our Video-3D LLM aligns video representations with real-world spatial contexts more accurately.

News

• [2025-3-4] We release the checkpoint of Video-3D LLM.
• [2025-2-27] Our paper is accepted to CVPR 2025.
• [2024-12-11] We release the processed data of Video-3D LLM.
• [2024-12-3] We release the paper of Video-3D LLM.

TODO List

• Release the training code.
• Release the evaluation script.
• Release the processed data.
• Release the model checkpoint.

Installation

1. Clone this repository and navigate to the Video-3D-LLM:

git clone https://github.com/LaVi-Lab/Video-3D-LLM.git
cd Video-3D-LLM

2. Create the conda environment:

conda create -n video3dllm python=3.10 -y
conda activate video3dllm
pip install --upgrade pip  # Enable PEP 660 support.
pip install -e ".[train]"
pip install flash-attn --no-build-isolation     # install flash attention

Data Preparation

For data preparation, please refer to this instruction for detail.

Training & Inference

Full-finetuned Training

Our model is fully fine-tuned using a combination dataset from five tasks, including ScanRefer, Multi3DRefer, SQA3D, ScanQA, Scan2Cap. The option frame_sampling_strategy in train_multi.sh should be set to one of the following options: ['uniform', 'mc-ratio90', 'mc-ratio95'], and the option frames_upbound determines the maximum number of frames used during training phase.

sh scripts/3d/train/train_multi.sh

Evaluation

Evaluate model performance with specified checkpoint and frame sampling strategy.

# Usage: sh scripts/3d/eval/eval_scan2cap.sh <CKPT_NAME> <SAMPLING_STRATEGY> <MAX_FRAMES>
# Parameters:
#   CKPT_NAME:           Path to model checkpoint for evaluation
#   SAMPLING_STRATEGY:   Frame sampling strategy (options: uniform, ratio90, ratio95)
#     - uniform:         Uniform frame sampling (equally spaced frames)
#     - ratio90:         Adaptive sampling until covering 90% of scene voxels
#     - ratio95:         Adaptive sampling until covering 95% of scene voxels
#   MAX_FRAMES:          Maximum number of frames to use (upper sampling limit)
# Note: Sampling strategies work consistently across tasks. For example:
#   sh scripts/3d/eval/eval_scan2cap.sh $CKPT uniform 32
sh scripts/3d/eval/eval_scan2cap.sh $CKPT_NAME $SAMPLING_STRATEGY $MAX_FRAMES   

Acknowledgements

We would like to thank the following works for their contributions to the opensourced codebase and community!

• LLaVA-Next: the codebase our codebase is built upon.
• LLaVA-3D, LEO, EmbodiedScan: We refer to these works for the data processing and evaluation setup.
• ScanNet, ScanRefer, Multi3DRefer, SQA3D, ScanQA: the datasets we use.

Citation

If you find our Video-3D LLM useful for your research, please consider giving this repository a star and citing our paper as follows:

@misc{zheng2024video3dllmlearningpositionaware,
      title={Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding}, 
      author={Duo Zheng and Shijia Huang and Liwei Wang},
      year={2024},
      eprint={2412.00493},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.00493}, 
}