CARL: Congestion-Aware Reinforcement Learning for Imitation-based Perturbations in Mixed Traffic Control

Paper accepted to IEEE CYBER 2024 (ArXiV)

Our RVs in the Ring

---

This work was done on top of FLOW framework obtained on Jan 3, 2023.

Installation instructions

Developed and tested on Ubuntu 18.04, Python 3.7.3

• Install Anaconda
• Setup SUMO versio 1.15.0
• Clone this repository
• Use the following commands to complete setup.

conda env create -f environment.yml
conda activate flow
python setup.py develop
pip install -U pip setuptools
pip install -r requirements.txt

---

Part 1: Training the Imitation Learning model on I24 Motion dataset

• For the data processing (car-following filter), follow the Notebook: Car-Following trajectory analysis
• Follow the Notebook: Imitation Learning

---

Part 2: Training the Congestion Stage Classifier

• Follow the Notebook: Ring

If you want to use the trained CSCs, see Data section below.

---

Part 3: Training RL based RVs

Go to the folder ring/Ours and enter the command:

# Train our policy in Ring (at 5% penetration) 
python train.py singleagent_ring

# Train our policy in Ring (at penetrations >5%)
python train.py multiagent_ring

---

Part 4: Generate rollouts for RL based RVs or Heuristic and Model based RVs and save as csv files.

All scripts related to this part are consolidated in the Notebook: Evaluate Ring

I. RL based RVs:

Replace the method name to be one of: ours, wu

python test_rllib.py [Location of trained policy] [checkpoint number] --method wu --gen_emission --num_rollouts [no_of_rollouts] --shock --render --length 260

II. Heuristic and Model based RVs:

For all (replace the method_name to be one of: bcm, lacc, piws, fs, idm)

python classic.py --method [method_name] --render --length 260 --num_rollouts [no_of_rollouts] --shock --gen_emission

For stability tests where a standard perturbation is applied by a leading HV, include --stability to the line above

---

Part 5: Evaluate the generated rollouts

To evaluate the generated rollouts into Safety and Efficiency metrics, replace the method name to be one of: idm, bcm, fs, piws, lacc, wu, ours.

python eval_metrics.py --method [method_name] --num_rollouts [no_of_rollouts]

To add plots to the metrics, include --save_plots

For Stability plots

python eval_plots.py --method [method_name]

---

Data

• Trained Policies (at all penetration rates):

  • Ours
  • Wu

• Trained CSC Models: HuggingFace

• Experiment Data (including rollouts and data for plots): HuggingFace

---

Cite

@article{Poudel2024Carl,
  title = {CARL: Congestion-Aware Reinforcement Learning for Imitation-based Perturbations in Mixed Traffic Control},
  author = {Poudel, Bibek and Li, Weizi and Li, Shuai},
  journal = {International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER)},
  year = {2024},
}