Add seeding RL examples folder (#178)

Add seeding RL examples folder (#178)

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: harshmahesheka

README.md

@@ -19,6 +19,7 @@ $ pip install -U -e git+https://github.com/projectmesa/mesa-examples@mesa-2.x#eg
 ```
 ```python
 from mesa_models.boltzmann_wealth_model.model import BoltzmannWealthModel
+
 ```
 You can see the available models at [setup.cfg](https://github.com/projectmesa/mesa-examples/blob/main/setup.cfg).
 

pyproject.toml

@@ -18,6 +18,12 @@ test_gis = [
     "pytest",
     "momepy",
 ]
+rl_example = [
+    "stable-baselines3",
+    "seaborn",
+    "mesa",
+    "tensorboard"
+]
 
 [build-system]
 requires = [

rl/.gitignore

@@ -0,0 +1 @@
+__pycache__/

rl/README.md

@@ -0,0 +1,66 @@
+# Reinforcement Learning Implementations with Mesa
+
+This repository demonstrates various applications of reinforcement learning (RL) using the Mesa agent-based modeling framework.
+
+<p align="center">
+<img src="wolf_sheep/resources/wolf_sheep.gif" width="500" height="400">
+</p>
+
+## Getting Started
+
+### Installation
+
+*Given the number of dependencies required, we recommend starting by creating a Conda environment or a Python virtual environment.*
+1. **Install Mesa Models**
+   Begin by installing the Mesa models:
+
+   ```bash
+   pip install -U -e git+https://github.com/projectmesa/mesa-examples@mesa-2.x#egg=mesa-models
+   ```
+
+2. **Install RLlib for Multi-Agent Training**
+   Next, install RLlib along with TensorFlow and PyTorch to support multi-agent training algorithms:
+
+   ```bash
+   pip install "ray[rllib]" tensorflow torch
+   ```
+
+3. **Install Additional Dependencies**
+   Finally, install any remaining dependencies:
+
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+4. **Download Pre-Trained Weights**
+   Download pre-trained weights from hugging face:
+
+   ```bash
+   git clone https://huggingface.co/projectmesa/rl_models/
+   ```
+
+### Running the Examples
+
+To test the code, simply execute `example.py`:
+
+```bash
+python example.py
+```
+
+*Note: Pre-trained models might not work in some cases because of differnce in versions of libraries used to train and test.*
+
+To learn about individual implementations, please refer to the README files of specific environments.
+
+
+## Tutorials
+
+For detailed tutorials on how to use these implementations and guidance on starting your own projects, please refer to [Tutorials.md](./Tutorials.md).
+
+Here's a refined version of your contribution guide:
+
+
+## Contribution Guide
+
+We welcome contributions to our project! A great way to get started is by implementing the remaining examples listed in the [Mesa-Examples](https://github.com/projectmesa/mesa-examples) repository with reinforcement learning (RL).
+
+Additionally, if you have your own Mesa environments that you think would benefit from RL integration, we encourage you to share them with us. Simply start an issue on our GitHub repository with your suggestion, and we can collaborate on bringing it to life!

rl/Tutorials.ipynb

@@ -0,0 +1,174 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Tutorial: Reinforcement Learning with Mesa Environments\n",
+    "\n",
+    "# Welcome to this comprehensive guide on integrating reinforcement learning (RL) with Mesa environments. \n",
+    "# Mesa, an agent-based modeling framework, offers an excellent platform to experiment with RL algorithms. \n",
+    "# In this tutorial, we'll explore several examples of how RL can be applied to various Mesa environments, \n",
+    "# starting with the **Epstein Civil Violence model**.\n",
+    "\n",
+    "# ## Getting Started\n",
+    "\n",
+    "# Before diving into the implementation, take a moment to familiarize yourself with the Epstein Civil Violence model.\n",
+    "# This will give you a solid understanding of the environment we’ll be working with.\n",
+    "\n",
+    "# Next, ensure all dependencies are installed by following the instructions in the `README.md`.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ### Step 1: Importing the Necessary Modules\n",
+    "# To begin, let’s import the required modules for the Epstein Civil Violence model:\n",
+    "\n",
+    "from epstein_civil_violence.model import EpsteinCivilViolenceRL\n",
+    "from epstein_civil_violence.server import run_model\n",
+    "from epstein_civil_violence.train_config import config\n",
+    "from train import train_model\n",
+    "\n",
+    "# Here’s a breakdown of the modules:\n",
+    "# - `EpsteinCivilViolenceRL`: Contains the core model and environment.\n",
+    "# - `run_model`: Configures and runs the model for inference.\n",
+    "# - `config`: Defines the parameters for training the model.\n",
+    "# - `train_model`: Includes functions for training the RL agent using RLlib."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ### Step 2: Initializing the Environment\n",
+    "\n",
+    "# Let's load and reset the environment. This also allows us to inspect the observation space:\n",
+    "\n",
+    "env = EpsteinCivilViolenceRL()\n",
+    "observation, info = env.reset(seed=42)\n",
+    "\n",
+    "# Display initial observation and info\n",
+    "print(\"Initial Observation:\", observation)\n",
+    "print(\"Info:\", info)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ### Step 3: Running the Environment with Random Actions\n",
+    "\n",
+    "# To get a feel for how the environment operates, let's run it for a few steps using random actions.\n",
+    "# We’ll sample the action space for these actions:\n",
+    "\n",
+    "for _ in range(10):\n",
+    "    action_dict = {}\n",
+    "    for agent in env.schedule.agents:\n",
+    "        action_dict[agent.unique_id] = env.action_space.sample()\n",
+    "    observation, reward, terminated, truncated, info = env.step(action_dict)\n",
+    "\n",
+    "    print(\n",
+    "        f\"Observation: {observation}, Reward: {reward}, Terminated: {terminated}, Truncated: {truncated}, Info: {info}\"\n",
+    "    )\n",
+    "\n",
+    "    if terminated or truncated:\n",
+    "        observation, info = env.reset()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ### Step 4: Training the Model\n",
+    "\n",
+    "# Now that you're familiar with the environment, let's train the RL model using the preset configuration:\n",
+    "\n",
+    "train_model(\n",
+    "    config, num_iterations=1, result_path=\"results.txt\", checkpoint_dir=\"checkpoints\"\n",
+    ")\n",
+    "\n",
+    "# You can modify the training parameters in the `train_config.py` file to experiment with different outcomes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ### Step 5: Visualizing the Results\n",
+    "\n",
+    "# After training, you can visualize the results by running inference on the model.\n",
+    "# Mesa's built-in visualization tools will help you launch a webpage to view the model's performance:\n",
+    "\n",
+    "# server = run_model(path=\"checkpoints\")\n",
+    "# You can also try running pre-trained checkpoints present in model folder\n",
+    "server = run_model(model_path=\"rl_models/epstein_civil_violence\")\n",
+    "server.port = 6005\n",
+    "server.launch(open_browser=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# ### Alternative Approach: Using Stable-Baselines with Mesa\n",
+    "\n",
+    "# In the example above, we utilized RLlib to integrate reinforcement learning algorithms with the Mesa environment, \n",
+    "# which is particularly useful when you want different policies for different agents. \n",
+    "# However, if your use case requires a simpler setup where all agents follow the same policy, \n",
+    "# you can opt for Stable-Baselines. An example of integrating Stable-Baselines with Mesa can be found in the Boltzmann Money model.\n",
+    "\n",
+    "# You can explore more on how to use Stable-Baselines with Mesa by following the respective documentation.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# ### Implementing Your Own Cases\n",
+    "\n",
+    "# If you're ready to explore RL in different agent-based scenarios, you can start by experimenting with various examples we provide at Mesa-Examples:\n",
+    "# Link: https://github.com/projectmesa/mesa-examples\n",
+    "\n",
+    "# These examples cover a range of scenarios and offer a great starting point for understanding how to apply RL within Mesa environments.\n",
+    "\n",
+    "# If you have your own scenario in mind, you can create it as a Mesa model by following this series of Tutorials:\n",
+    "# Link: https://mesa.readthedocs.io/en/stable/tutorials/intro_tutorial.html\n",
+    "\n",
+    "# Once your scenario is set up as a Mesa model, you can refer to the code in the provided implementations to see how the RL components are built on top of the respective Mesa models.\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "test",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

rl/epstein_civil_violence/README.md

@@ -0,0 +1,30 @@
+# Modelling Violence: Epstein Civil Violence Model
+
+This project demonstrates the use of the RLlib library to implement Multi-Agent Reinforcement Learning (MARL) in the classic Epstein-Civil Violence problem. The environment details can be found on the Mesa project's GitHub repository [here](https://github.com/projectmesa/mesa-examples/tree/main/examples/epstein_civil_violence).
+
+## Key Features
+
+**RLlib and Multi-Agent Learning**:
+- **Library Utilized**: The project leverages the RLlib library to concurrently train two independent PPO (Proximal Policy Optimization) agents.
+- **Agents**:
+  - **Police**: Aims to control violence (Reduce active agent)
+  - **Citizen**: Aims to show resistence (be active) without getting arrested
+
+**Input and Observation Space**:
+- **Observation Grid**: Each agent's policy receives a 4 radius grid centered on itself as input.
+
+**Action Space**:
+- **Action Space**: For citizen the action space is the ID of the neighboring tile to which the agent wants to move along with choice to be active. For cop the action space is ID of neighbourng tile it wants to move along with ID of active citizen in it's neigbhood that it wants to arrest.
+**Behavior and Training Outcomes**:
+
+**Optimal Behavior**:
+  - **Cops**: Learns to move towards active agents and arrest them.
+  - **Citizens**: Learns to run away from cops and be active only if a cop isn't around.
+- **Density Variations**: You can vary the densities of sheep and wolves to observe different results.
+
+By leveraging RLlib and Multi-Agent Learning, this project provides insights into the dynamics of violence in a society and various strategies in a simulated environment.
+
+
+<p align="center">
+<img src="resources/epstein.gif" width="500" height="400">
+</p>

rl/epstein_civil_violence/agent.py

@@ -0,0 +1,55 @@
+from mesa_models.epstein_civil_violence.agent import Citizen, Cop
+
+from .utility import move
+
+
+class CitizenRL(Citizen):
+    def step(self):
+        # Get action from action_dict
+        action_tuple = self.model.action_dict[self.unique_id]
+        # If in jail decrease sentence, else update condition
+        if self.jail_sentence > 0:
+            self.jail_sentence -= 1
+        else:
+            # RL Logic
+            # Update condition and postion based on action
+            self.condition = "Active" if action_tuple[0] == 1 else "Quiescent"
+            # Update neighbors for updated empty neighbors
+            self.update_neighbors()
+            if self.model.movement:
+                move(
+                    self,
+                    action_tuple[1],
+                    self.empty_neighbors,
+                )
+
+        # Update the neighbors for observation space
+        self.update_neighbors()
+
+
+class CopRL(Cop):
+    def step(self):
+        # RL Logics
+        # Arrest if active citizen is indicated in action
+        action_tuple = self.model.action_dict[self.unique_id]
+        arrest_pos = self.neighborhood[action_tuple[0]]
+        for agent in self.model.grid.get_cell_list_contents(self.neighborhood):
+            if (
+                isinstance(agent, CitizenRL)
+                and agent.condition == "Active"
+                and agent.jail_sentence == 0
+                and agent.pos == arrest_pos
+            ):
+                agent.jail_sentence = self.random.randint(1, self.model.max_jail_term)
+                agent.condition = "Quiescent"
+                self.arrest_made = True
+                break
+            else:
+                self.arrest_made = False
+        # Update neighbors for updated empty neighbors
+        self.update_neighbors()
+        # Move based on action
+        if self.model.movement:
+            move(self, action_tuple[1], self.empty_neighbors)
+        # Update the neighbors for observation space
+        self.update_neighbors()