SegNet4D: Efficient Instance-Aware 4D Semantic Segmentation for LiDAR Point Cloud

This repo contains the implementation of our SegNet4D, which is an extension of our conference paper, i.e., InsMOS. SegNet4D is a new LiDAR-only SOTA method for 4D semantic segmentation with public paper (as of October 2024).

SegNet4D: Efficient Instance-Aware 4D Semantic Segmentation for LiDAR Point Cloud

Neng Wang, Ruibin Guo, Chenghao Shi, Ziyue Wang, Hui Zhang, Huimin Lu, Zhiqiang Zheng, Xieyuanli Chen

News:

• [2025-05] Our paper is accepted by T-ASE.

• [2025-04] Code released.

• [2024-11] Paper is available in arXiv website.

Framework

SegNet4D is an efficient Instance-Aware 4D LiDAR semantic segmentation framework. We first utilize the Motion Features Encoding Module to extract motion features from the sequential LiDAR scans. Following this, the motion features are concatenated with the spatial features of the current scan and fed into the Instance-Aware Feature Extraction Backbone. Then, two separate heads are applied: a motion head for predicting moving states, and a semantic head for predicting semantic category. Finally, the Motion-Semantic Fusion Module integrates the motion and semantic features to achieve 4D semantic segmentation.

Related Video

Our accompanying video is now available on OneDrive.

Data

We mainly train our model on the SemanticKITTI and nuScenes dataset.

1. SemanticKITTI

Download the raw LiDAR scan dataset from KITTI website and semantic annotations from SemanticKITTI website.

generating instance bounding box:

python utils/generate_boundingbox.py --data_path ./demo_data/ --view --lshape --save

--data_path: data path --view: Visualizing the instance box

--lshape: using the L-shap for refining the box --save: saving the box in the .npy file.

Before running this, you need to install open3d and PCL in python environment.

You can download the bounding box from the link directly.

Data structure

└── sequences
  ├── 00/           
  │   ├── velodyne/	
  |   |	├── 000000.bin
  |   |	├── 000001.bin
  |   |	└── ...
  │   ├── labels/ 
  |   | ├── 000000.label
  |   | ├── 000001.label
  |   | └── ...
  |   ├── boundingbox_label
  |   | ├── 000000.npy
  |   |	├── 000001.npy
  |   |	└── ...
  |   ├── residual_bev_images_1
  |   ├── residual_bev_images_2
  |   ├── residual_bev_images_3...
  |   ├── calib.txt
  |   ├── poses.txt
  |   └── times.txt
  ├── 01/ # 00-10 for training
  ├── 08/ # for validation
  ├── 11-21/ # 11-21 for testing

2. nuScenes

Download the raw dataset from the website.

generating nuScenes multi-scan dataset

You can find detailed readme here.

Code usage

1. Installation

system dependencies:

ubuntu20.04, CUDA 11.3, cuDNN 8.2.1, 

python dependencies:

python 3.7

2. Environments

conda create --name segnet4d python=3.7
conda activate segnet4d
pip install -r requirements.txt

# insltall pytorch with cuda11.3, avoid using "pip install torch"
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge

# ensure numpy==1.18.1
pip uninstall numpy
pip install numpy==1.18.1

install segnet4d

git clone https://github.com/nubot-nudt/SegNet4D.git
cd SegNet4D

# install sptr library (details form the readmed in this directory)
cd models/SparseTransformer
python3 setup.py install

# install SegNet4D
cd SegNet4D
python setup.py develop

3. Running

Generating BEV residual images

If you want to train quickly, you need to generate BEV residual in advance.

Note that this is only required for the SemanticKITTI dataset. For the nuScenes dataset, you can generate the BEV residual online.

python utils/gen_residual_bev.py

Training

# nuscenes
export DATA=/path/to/nuScenes_kitti 
python scripts/train_nuscenes.py

# semantickitti
export DATA=/path/to/kitti/sequences
python scripts/train.py  

Inference

You can download our ckpt from this link.

# demo data
python scripts/predict_nuscenes.py --cfg_file config/nuscenes/nuscenes_config.yaml --data_path /demo_data/nuScenes_kitti/ --ckpt ./ckpt/nuscenes.ckpt --split demo

# nuscenes
python scripts/predict_nuscenes.py --cfg_file config/nuscenes/nuscenes_config.yaml --data_path /home/wangneng/DataFast/nuScences_kitti --ckpt ./ckpt/nuscenes.ckpt

---

Visualization

Run the following command to visualize the results of multi-scan semantic predictions and mos predictions.

Press key n to show next frame

Press key b to show last frame

Press key q to quit display

cd visualization
python vis_sem.py
python vis_mos.py

Evaluation

python eval/evaluate_semantics_nuscenes.py --dataset /home/wangneng/DataFast/nuScences_kitti --predictions preb_out/SegNet4D/multi_semantic_preb/

python eval/evaluate_mos_nuscenes.py --dataset /home/wangneng/DataFast/nuScences_kitti --predictions preb_out/SegNet4D/mos_preb/

Citation

If you use our code in your work, please star our repo and cite our paper.

@inproceedings{wang2023iros,
	title={{InsMOS: Instance-Aware Moving Object Segmentation in LiDAR Data}},
	author={Wang, Neng and Shi, Chenghao and Guo, Ruibin and Lu, Huimin and Zheng, Zhiqiang and Chen, Xieyuanli},
	booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
    pages={7598-7605},
	year={2023}
}

@article{wang2024arxiv,
	title={{SegNet4D: Efficient Instance-Aware 4D Semantic Segmentation for LiDAR Point Cloud}},
	author={Wang, Neng and Guo, Ruibin and Shi, Ziyue Wang, Chenghao and Zhang, Hui and Lu, Huimin and Zheng, Zhiqiang and Chen, Xieyuanli},
	journal={arXiv preprint},
	year={2024}
}

Contact

Any question or suggestions are welcome!

Neng Wang: nwang@nudt.edu.cn and Xieyuanli Chen: xieyuanli.chen@nudt.edu.cn

Acknowledgment

We thank for the opensource codebases, MapMOS, AutoMOS