Repository for Online Fine-Tuning Segmenting Decision Transformer (SegDT) for Semantic Object Goal-Reaching task

Installation

We tested SegDT with Python 3.10 and CUDA 11.8.

1. Create conda environment.

conda create -n habitat python=3.10 cmake=3.14.0 -y
conda run -n habitat pip install numpy==1.23.5 gym==0.22.0
conda run -n habitat pip install numba==0.58.0
conda run -n habitat pip install numpy-quaternion==2023.0.0

2. Install habitat-sim.

git clone --branch challenge-2023 https://github.com/facebookresearch/habitat-sim.git
cd habitat-sim 
conda run -n habitat python setup.py install --headless --bullet --with-cuda

3. Install torch.

conda install -n habitat pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=11.8 -c pytorch -c nvidia

4. Install habitat-lab.

git clone --branch challenge-2023 https://github.com/facebookresearch/habitat-lab.git
cd habitat-lab/home-robot; conda run -n habitat pip install src/home_robot
conda run -n habitat pip install opencv-python 
conda install -n habitat numpy==1.23.5
cd habitat-lab/habitat-lab; conda run -n habitat python setup.py develop --all --allow-hosts 

5. Install other packages.

conda run -n habitat pip install -r requirements.txt
conda run -n habitat pip install git+https://github.com/openai/CLIP.git
conda run -n habitat pip install timm transformers
conda run -n habitat pip install 'git+https://github.com/facebookresearch/fvcore'

6. Install detectron2

conda install -n habitat nvidia/label/cuda-11.8.0::cuda-toolkit
conda install -n habitat nvidia/label/cuda-11.8.0::cuda-nvcc
export CUDA_HOME=/home/<PATH-TO-CONDA-ENV>/envs/habitat
conda run -n habitat pip install 'git+https://github.com/facebookresearch/detectron2'

Data

We provide GoalReacher episodes in the datasets/ folder. datasets/train_scenes_goalreacher contains episodes for training scens, where as datasets/val_scenes_goalreacher contains episodes for validation scenes.

Training

1. Training from Scratch with RNN-based SegDT Sampling and Transformer-based (NanoGPT) SegDT Training with AWR Loss and Segmentation loss

python RUN_Train_RNN_PPO_AWR.py --GPU 0 --RNN True --segm_loss True

2. Training from Scratch with RNN-based SegDT Sampling and Transformer-based (NanoGPT) SegDT Training with AWR Loss without Segmentation loss

python RUN_Train_RNN_PPO_AWR.py --GPU 0 --RNN True --segm_loss False

Validation of SegDT on GoalReacher task

0. Unzip pretrained weights SegDT_TRAIN_nL_SegDT_RLclone_SemFull_8903_1.ckpt.zip, do not change the weights path.

1. Run SegDT with GT goal segmentation:

python eval_gt_segmentation.py 

1. Run SegDT with predicted goal segmentation:

python eval_pred_segmentation.py