1.5.0

Discrete MASAC rewrite, static seeding in evaluation, experiment relaoding, tutorials, task restyle and much more

BenchMARL release paired with TorchRL 0.8

BC-Breaking changes:

• Decouple Task enums from task classes by @matteobettini in #175 -> before, tasks were an enum that also contained the logic to create an environment, now the two concepts are separate. Task just contains the enum of available tasks and TaskClass contains the environment creation logic. This has been made to address #173

[Tutorial] Creating a VMAS scenario and training it #172

  Creating a VMAS scenario and training it in BenchMARL. We will create a scenario where multiple robots with different embodiments need to navigate to their goals while avoiding each other (as well as obstacles) and train it using MAPPO and MLP/GNN policies.

[Feature] Reload experiment without re-passing config #179

You can now reload the experiment in many ways just by passing the checkpoint file
https://benchmarl.readthedocs.io/en/latest/concepts/features.html#reloading

[Feature, BC-breaking] Decouple Task enums from task classes #175

before, tasks were an enum that also contained the logic to create an environment, now the two concepts are separate. Task just contains the enum of available tasks and TaskClass contains the environment creation logic. This has been made to address #173

Example:

VmasTask.BALANCE # Task
VmasTask.BALANCE.get_from_yaml(path=None) # TaskClass, you can pass a custom yaml file path
VmasTask.BALANCE.get_task(config=None)  # TaskClass, you can pass a custom config if you don't want to load from yaml
assert  VmasTask.BALANCE.get_from_yaml()  == VmasTask.BALANCE.get_task(config=None)

[Feature] Static evaluation #185

By setting experiment.evaluation_static=True evaluation environments will be the same throughout training and in between runs

[Feature] Discrete MASAC rewrite #192

#192 has a really in depth explanation.

TL;DR look how good are the new Discrete MASAC implementations wrt to the old one on VMAS balance

What's Changed

• [Tutorial] Creating a VMAS scenario and training it by @matteobettini in #172
• [Feature] Usability improvement in ExperimentConfig by @itwasabhi in #174
• Ensure experiment uses user configured buffer device for replay buffer by @itwasabhi in #180
• [Feature] Reload experiment without re-passing config by @matteobettini in #179
• [Minor] Seeding and logging by @matteobettini in #183
• [Docs] Section on rendering and devices by @matteobettini in #184
• [Feature, BC-breaking] Decouple Task enums from task classes by @matteobettini in #175
• [Feature] Static evaluation by @matteobettini in #185
• [Feature] Pass devices to exploration modules by @matteobettini in #191
• [Feature] Discrete MASAC by @matteobettini in #192
• [Minor] Add kwargs to Plotting methods by @Xmaster6y in #195
• [Fine-tune] Restore VMAS gamma by @matteobettini in #198

New Contributors

• @itwasabhi made their first contribution in #174
• @Xmaster6y made their first contribution in #195

Full Changelog: 1.4.0...1.5.0

1.4.0

BenchMARL release paired with TorchRL 0.7.0

A lot of new features and many bug fixes!

New environment: MAgent2

Thans a lot to @JoseLuisC99 for making their first contribution and helping in this effort

• [Environment] MAgent2 by @JoseLuisC99 in #137

Parallel collection for non-vectorized envs

Thanks to a lot of help from @gliese876b we have introduced parallel collection for non-vectorized envs using torchrl ParallelEnv

Check it out here:

• [Feature] Parallel collection by @matteobettini in #152

Prioritized replay buffers

Again, thanks a mil to @gliese876b

• [Feature] Prioritized Replay Buffer by @gliese876b in #160

Ensemble models and algorithms

From the amazing request of @karthiks1701 we have introduced the possibilty of using different models and algorithms for different agent groups

Check it out here: https://benchmarl.readthedocs.io/en/latest/concepts/features.html#ensemble-models-and-algorithms

• [Feature] Ensemble models and algorithms (different chioces for different agent groups) by @matteobettini in #159

Disk storage for replay buffers

Again, thanks to the input of @JoseLuisC99, we have added the possibility to store buffers on diisk

• [Feature] Disk storage for replay buffers by @JoseLuisC99 in #155

VMAS football

Enjoy the full power of the new VMAS football enviornment in BenchMARL

More info here: https://github.com/proroklab/VectorizedMultiAgentSimulator/releases/tag/1.5.0

What's Changed

• [Errors] Error on unavailable combinations by @matteobettini in #142
• [BugFix] Fix meltingpot buffer transforms by @matteobettini in #148
• [BugFix] VMAS parameters names by @matteobettini in #149
• [Environment] MAgent2 by @JoseLuisC99 in #137
• [Docs] MAgent by @matteobettini in #150
• [Minor] Dependencies by @matteobettini in #151
• [Feature] Parallel collection by @matteobettini in #152
• [Feature] Prioritized Replay Buffer by @gliese876b in #160
• [Docs] MAgent typo by @matteobettini in #162
• [Feature] Ensemble models and algorithms (different chioces for different agent groups) by @matteobettini in #159
• [Feature] Disk storage for replay buffers by @JoseLuisC99 in #155
• [BugFix] Tensorboard hparam logging by @matteobettini in #165
• [TorchRL deprecation] AdditiveGaussianWrapper -> AdditiveGaussianModule by @matteobettini in #167
• [Task] VMAS football by @matteobettini in #166
• [Fine-tune] Update gammas by @matteobettini in #168
• [Dependency] Make video rendering deps core by @matteobettini in #169
• [Release] 1.4.0 by @matteobettini in #170

New Contributors

• @JoseLuisC99 made their first contribution in #137
• @gliese876b made their first contribution in #160

Full Changelog: 1.3.0...1.4.0

1.3.0

Memory networks in BenchMARL

BenchMARL release paired with TorchRL 0.6.0

Highlights

RNNs in BenchMARL

We now support RNNs as models!

We have implemented and added to the library GRU and LSTM.

These can be used in the policy or in the critic (both from local agent inputs and from global centralized inputs). They are also compatible with any parameter sharing choice.

We have benchmarked GRU on a multiagent version of repeat_previous and it does solve the task, while just MLP fails

In cotrast to traditional RNNs, we do not do any time padding. Instead we do a for loop over time in the sequence while reading the “is_init” flag. This approach is slower but leads to better results.

Oh also as always you can chain them with as many models as you desire (CNN, GNN, ...)

Simplified model reloading and evaluation

We have added some useful tools described at here

In particular, we have added experiment.evaluate() and some useful command line tols like benchmarl/evaluate.py and benchmarl/resume.py that just take the path to a checkpoint file.

Basically now you can reload models from the hydra run without giving again all the config, the scripts will find the configs you used in the hydra folders automatically.

Better logging of episode metrics

BenchaMARL will now consider an episode done when the global task done is set. Thus, it will allow for agents done early (as long as the global done is set on all()

Here is an overview of how episode metrics are computed:

BenchMARL will be looking at the global done (always assumed to be set), which can usually be computed using any or all over the single agents dones.

In all cases the global done is what is used to compute the episode reward.

We log episode_reward min, mean, max over episodes at three different levels:

• agent, (disabled by default, can be turned on manually)
• group, averaged over agents in group
• global, averaged over agents in groups and gropus

What's Changed

• [BugFix] Fix collect with grad by @matteobettini in #114
• [Refactor] Update values headed to deprecation by @matteobettini in #118
• [Docs] Citation by @matteobettini in #119
• [Model] GRU and general RNN compatibility by @matteobettini in #116
• [Model] LSTM by @matteobettini in #120
• [BugFix, Feature] GNN position_key and velocity_key not in observation_spec by @matteobettini in #125
• [BugFix] Small fix to multi-group eval and add wandb project_name by @matteobettini in #126
• [Feature] Improved experiment reloading and evaluation by @matteobettini in #127
• [BugFix] GNN position and velocity key by @matteobettini in #132
• [BugFix] More flexible episode_reward computation in logger by @matteobettini in #136
• [Feature] RNNs in SAC losses by @matteobettini in #139
• [Version] 1.3.0 by @matteobettini in #140

Full Changelog: 1.2.1...1.3.0

1.2.1

BenchMARL release paired with TorchRL 0.4.0

Many exctiting updates in this new release!

Most importantly there have been major inmpovements to the GNN and the addition of a new model: DeepSets

New features

• [Feature] Allow multiple observation keys in specs by @matteobettini in #82 #83
• [Feature] Buffer device by @matteobettini in #87
• [Feature] More VMAS tasks by @matteobettini in #88
• [Feature] Simplify extending benchmarl and improve extending examples by @matteobettini in #89
• [Example] Collecting with gradient by @matteobettini in #77
• [Feature] Share parameters between models by @matteobettini in #95
• [Feature] GNN build topology dynamically from positions by @matteobettini in #98
• [Model] DeepSets by @matteobettini in #96
• [Feature] PPO minibatch advantage by @matteobettini in #100
• [Feature] keep_checkpoints_num and checkpoint_at_end by @matteobettini in #102
• [Feature] Run evaluation on first iteration by @matteobettini in #108
• [Feature] Iteration timer includes evaluation by @matteobettini in #110

New Contributors

• @mchoilab made their first contribution in #105

Full Changelog: 1.2.0...1.2.1

1.2.0

BenchMARL release paired with TorchRL 0.4.0

BenchMARL just got extended with 49 vision-based tasks from Google DeepMind MeltingPot 🎉

But wait there is more! We have added a brand new CNN model 🚀 that can be used in actors and critics (fully compatible with any parameter sharing and centralization choices)

What's Changed

• [Feature] Allow any name for observation and global_state keys by @matteobettini in #72
• [Model] CNN by @matteobettini in #74
• [Feature] Allow multipe inputs to models by @matteobettini in #73
• [Environment] Melitngpot by @matteobettini in #75

Full Changelog: 1.1.1...1.2.0

1.1.1

Align to torchrl 0.3.1

What's Changed

• [CI] Fix CI by @matteobettini in #65
• [Test] Lighter tests by @matteobettini in #66
• [CI] Fix CI by @matteobettini in #67
• [Release] 1.1.1 by @matteobettini in #69

Full Changelog: 1.1.0...1.1.1

1.1.0

Highlights

• New GNN Model
• New VMAS scenarios and VMAS MPE scenarios
• Multiple bug fixes and new configuration parameters
• BenchMARL 1.1.0 is paired to TorchRL 0.3.0

What's Changed

• [Model] GNN by @matteobettini in #30
• [Docs] Citation by @matteobettini in #39
• [BugFix] Fix discrete SAC by @matteobettini in #40
• [Tasks] VMAS MPE (Vectorized) by @matteobettini in #42
• [Feature] Optional Tanh transfromed distributions in algorithms by @matteobettini in #45
• [Feature] Choose if evaluation is run on sampled actions or on mode by @matteobettini in #49
• [BugFix] Use correct types to prevent multiwalker init error. by @KaleabTessera in #47
• [CI] Fix code cov by @matteobettini in #51
• [Feature] More VMAS tasks by @matteobettini in #48
• [Doc] Fix Typo by @matteobettini in #57
• [BugFix] Rename PPO params by @matteobettini in #59
• [Doc] Fix typo by @matteobettini in #60
• [CI] Run main tests with torchrl nightly and have a separate CI for testing torchrl stable by @matteobettini in #53
• [Refactor] Adapt PPO parameter names by @matteobettini in #62
• [Versioning] 1.1.0 by @matteobettini in #61

New Contributors

• @KaleabTessera made their first contribution in #47

Full Changelog: 1.0.0...1.1.0

1.0.0

BenchMARL docs

The BenchMARL docs are now available at https://benchmarl.readthedocs.io/.

What's Changed

• [Feature] More callbacks by @matteobettini in #35
• [Docs] Link video presentation by @matteobettini in #36
• [BugFix] Task loading schema validation by @matteobettini in #37
• [Docs] Build the docs by @matteobettini in #34

Full Changelog: 0.0.4...1.0.0

0.0.4

0.0.4

0.0.3

• Added yaml configuration files to PyPi package
• Addded possibility to access the experiment from the algorithm