EgoMusic-driven Human Dance Motion Estimation with Skeleton Mamba

Quang Nguyen • Nhat Le • Baoru Huang • Minh Nhat Vu • Chengcheng Tang • Van Nguyen • Ngan Le • Thieu Vo • Anh Nguyen

_{^{ICCV 2025}}

Environment Preparation

Follow these steps to install the GraspMAS framework:

1. Clone repo:

   git clone https://github.com/Fsoft-AIC/SkeletonMamba.git
   cd SkeletonMamba

2. Prepare environment: cuda==11.8,python==3.11, torch==2.2.0, gcc==11.3 (for State Space Model enviroment). Installing Mamba may cost a lot of effort. If you encounter problems, this issues in Mamba may be very helpful.

Install virtual environment

bash env.sh

Install mujoco.

wget https://github.com/deepmind/mujoco/releases/download/2.1.0/mujoco210-linux-x86_64.tar.gz
tar -xzf mujoco210-linux-x86_64.tar.gz
mkdir ~/.mujoco
mv mujoco210 ~/.mujoco/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/.mujoco/mujoco210/bin

Install PyTorch3D.

conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install -c bottler nvidiacub
pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu113_pyt1110/download.html

3. Quick start:

model = SkeletonMamba(
        input_dim=9,
        embed_dim=128,
        depth=8,
        num_joints=24,
        has_text=True,
        d_context=128,
        device="cuda",
        use_pe=2,
        video_frames = 120,
    ).to("cuda")
x = torch.rand(10, 2880, 9).to("cuda")
t = torch.rand(10).to("cuda")
_context = torch.rand(10, 10, 128).to("cuda")
o = model(x, t, y=_context)
_param_count = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Param count: {_param_count}")
print(o.shape)
print(model)

Dataset preparation

The dataset will be published soon!!

Training and Eval

1. To train the model, run:

bash scripts/train.sh

2. To evaluate, run:

bash scripts/eval.sh

Citation

Please cite our paper:

@InProceedings{nguyen2025egomusic,
      title={EgoMusic-driven Human Dance Motion Estimation with Skeleton Mamba},
      author={{Nguyen, Quang and Le, Nhat and Huang, Baoru and Vu, Minh Nhat and Tang, Chengcheng and Nguyen, Van and Le, Ngan and Vo, Thieu and Nguyen, Anh},
      booktitle = {ICCV},
      year={2025}
}

Acknowledgement

The code base is develop and adapt from Zigma, EgoEgo