You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Here you can find all important changes in this project. RL-Studio adheres to [Semantic Versioning](http://semver.org/).
4
+
## [Unreleased]
5
+
6
+
## [1.2.0] - 2022-01-11
7
+
### Added:
8
+
9
+
- Add new files in envs/gazebo/f1/models/: original files are split into different modular files such as images.py, reset.py, step.py, settings.py, simplified_perception.py ([PR #122](https://github.com/JdeRobot/RL-Studio/pull/122))
10
+
11
+
### Changed:
12
+
- entry file name: `rl-studio.py `
13
+
- config file names: now they follow naming `config_mode_task_algorithm_agent_simulator`
14
+
- files structure in envs/gazebo/f1/models/: files are named `task_algorithm.py`
If you are contributing to RL-Studio tool development, please follow the below coding style and structure of directories, files recommendations:
4
+
5
+
## SOME RULES
6
+
7
+
-[PEP 8](https://peps.python.org/pep-0008/) style guide for Python Code
8
+
-[Black](https://github.com/psf/black) format
9
+
-[Pylint](https://pypi.org/project/pylint/) as static code analyser
10
+
- Constants variable must be upper-case (e.g `TIME_THRESHOLD`, `NUMBER_OF_RUNS`)
11
+
- Comment all non trivial functions to ease readability
12
+
- All the internal imported packages must be imported from the root of the project (e.g `import rl_studio.agents` instead `import agents`)
13
+
- Organize imports before pushing your code to the repo
14
+
15
+
- When creating a project, please keep in mind:
16
+
17
+
- in **/agents** directory, files names should be `mode_task_algorithm_simulator_framework.py`, i.e. `trainer_followline_ddpg_F1_gazebo_tf.py` or `inferencer_mountaincar_qlearn_openai_pytorch.py`. In case of not using framework leave it blank.
18
+
- in **/envs/gazebo/f1/models** directory, files names should be `task_algorithm_framework.py`, i.e. `followline_ddpg_gazebo_tf.py` or `followlane_qlearn_pytorch.py`. In case of not using framework leave it blank.
19
+
- As a general rule, **classes names** have to follow convention `ModeTaskAlgorithmAgentSimuladorFramework`, i.e. `TrainerFollowLaneDDPGF1GazeboPytorch` or `InferencerFollowLaneDQNF1GazeboTF`
20
+
- in **/envs/gazebo** directory, classes names follow rule `TaskAlgorithmAgentSimulatorFramework`, i.e. `FollowlineDDPGF1GazeboTF`.
21
+
22
+
# Directory architecture
23
+
24
+
## Config files
25
+
26
+
- in **/config** directory add a configuration file with the following format `config_mode_task_algorithm_agent_simulator.yaml`, i.e. `config_training_followlane_qlearn_F1_carla.yaml`
27
+
28
+
## Models
29
+
30
+
- Add a trained brain in **/checkpoints** folder. You can configure it in the config.yaml file. Automatically the app will add a directory with the format `task_algorithm_agent_simulator_framework` where to save models.
31
+
- The file model should have the format `timestamp_maxreward_epoch_ADITIONALTEXT.h5` in format h5 i.e. `09122002_max45678_epoch435_actor_conv2d32x64_critic_conv2d32x64_actionsCont_stateImg_rewardDiscrete.h5` to indicate the main features of the model saved in order to easily find the exact model.
32
+
33
+
## Metrics
34
+
35
+
- In **/metrics** folder should be saved statistics and metrics of models. You can configure it in the config.yaml file. Automatically the app will add a directory with the format `mode/task_algorithm_agent_simulator_framework/data` where to save data.
36
+
37
+
## Graphics
38
+
39
+
- In **/metrics** folder should be saved graphics of models. You can configure it in the config.yaml file. Automatically the app will add a directory with the format `mode/task_algorithm_agent_simulator_framework/graphics` where to save graphics.
40
+
41
+
## Logs and TensorBoard files
42
+
43
+
- In **/logs** folder should be saved TensorBoard and logs files. You can configure it in the config.yaml file.
44
+
For TensorBoard, automatically the app will add a directory with the format `mode/task_algorithm_agent_simulator_framework/TensorBoard`.
45
+
46
+
For logs, the app automatically will add a directory with the format `mode/task_algorithm_agent_simulator_framework/logs`.
47
+
48
+
# TIPS
49
+
50
+
- You can refer to "mountain_car qlearning" project as an example of how to implement a real time monitoring
51
+
- You can refer to "cartpole dqn" project as an example of how to implement logging
This file contains a set of rules to contributing to the project and the
6
+
rest of the projects developed by JdeRobot.
7
+
If you have any doubt about how to contribute contact one of the maintainers
8
+
of the project. They will be pleased to tell you how you can contribute with your
9
+
knowledge to the project and organization!
10
+
11
+
*[Code of conduct](#code-of-conduct)
12
+
*[Prerequisites before contributing](#prerequisites-before-contributing)
13
+
*[How can I contribute?](#how-can-i-contribute)
14
+
15
+
16
+
<aname="#code-of-conduct"></a>
17
+
## Code of conduct
18
+
Please report any unacceptable behavior to any of [the maintainers](#i-have-a-question).
19
+
20
+
<aname="#prerequisites"></a>
21
+
## Prerequisites before contributing
22
+
In order to contribute to JdeRobot projects, please read carefully the project README.md/[webpage](https://github.com/JdeRobot/RL-Studio) before
23
+
starting contributing to understand the purpose of the project and where you can contribute.
24
+
25
+
<aname="#how-to-contribute"></a>
26
+
## How can I contribute?
27
+
Any JdeRobot project follows the same workflow when contributing.
28
+
29
+
***Find a problem or possible improvement for the project:** First of all, check that the feature/bug is not listed in the current [open issues](https://github.com/JdeRobot/RL-Studio/issues).
30
+
31
+
***Create an issue:**[Create an issue](https://github.com/JdeRobot/RL-Studio/issues/new) in the project with the problem/improvement you will
32
+
address. In this issue, explain carefully what you will be updating and how this changes will impact the project.
33
+
Provide any complementary information to explain it (code samples, screenshots ...). You should information about:
34
+
* Expected behavior
35
+
* Actual behavior
36
+
* Steps to reproduce
37
+
* Environment
38
+
* Possible cause
39
+
* Possible solution
40
+
41
+
The two following points are different depending on the permissions you have to the repo.
42
+
***[If you have write permission] Work in a separate branch always:** Create a new branch with a describable name (you can use the issue number as branch name "issue_xxx"). Create your commits in that branch making the needed changes. Please, use describable names as commit messages, so everyone can understand easily the changes you made.
43
+
44
+
***[If you only have read permission] Fork the project:** Fork the project. Work on that copy of the repo, making the desirable changes. Please, use describable names as commit messages, so everyone can understand easily the changes you made.
45
+
46
+
***Open a pull request:** A pull request is compulsory any time a new change wants to be added to the core or the project. After solving the issue, create a pull request with your branch. In this pull request include all the commits made, write a good description of the changes made and refer to the issue solved to make things easier to the maintainers. Include any additional resource that would be interesting (references, screenshots...). Link the PR with the issue.
47
+
48
+
***Testing and merging pull requests**
49
+
One of maintainers will review your code. Reviewer could ask you to modify your pull request.
50
+
Please provide timely response for reviewers (within weeks, not months), otherwise you submission could be postponed or even rejected.
51
+
52
+
***[If you have write permission] Don't accept your own pull requests:** Wait for a project maintainer to accept the changes you made. They will probably comment the pull request with some feedback and will consider if it can be merge to the master branch. Be proactive and kind!
RL-Studio is a platform for training reinforcement learning algorithms for robots with different environments and algorithms. You can create your agent, environment and algorithm and compare it with others.
13
+
Reinforcement Learning Studio, RL-Studio, is a platform for developing robotic applications with reinforcement learning algorithms. Its modular design allows to work easily with different agents and algoritmhs in autonomous tasks and any simulator.
14
14
15
-
## Installation
15
+
#Introduction
16
16
17
-
### Install ROS
17
+
RL-Studio is designed to work with robots, as autonomous vehicles, in any relevant tasks with any simulators that provide adequate realism to transfer development to real environments automatically and with robustness --sim2real transfer.
18
+
The designed agents can support any type of sensor that collects information from the environment and, with reinforcement learning algorithms, can send correct signals to the actuators to adequately command the robot, following the standard reinforcement learning cycle.
18
19
19
-
RL-Studio works with ROS Noetic. You can [install ROS Noetic in the official documentation](http://wiki.ros.org/noetic/Installation/Ubuntu) and installing ROS Noetic Full Desktop.
20
+
## Working Modes
21
+
22
+
Rl-Studio allows you to work with different modes of operation, all of them neccesary to build a RL app:
23
+
24
+
- Training: the objective of any development in RL-Studio is to design a training that generates a suitable model for the environment in question. see diagram
25
+
- Retraining of models already generated, so that they continue learning in the same or different environments.
26
+
- Inference: Trained models are tested --inference -- in different environments in order to validate their learning.
27
+
28
+
## Agents
29
+
30
+
RL-Studio is designed to work with any robotic agent, mainly in autonomous driving through complex environments. However, thanks to the modularity of the application, it is easy to create new agents that are tested in other tasks, such as manipulation, legged robots, drones and so on.
31
+
32
+
## Algorithms
33
+
34
+
Qlearn, DQN, DDPG and PPO have currently been implemented to work on the different tasks developed. However, it is easy to design any other algorithms.
35
+
36
+
## Deep Learning frameworks
37
+
38
+
[Tensorflow](https://www.tensorflow.org/) 2.9.11 and [Pytorch](https://pytorch.org/) 1.13 are currently supported, although it is very easy to extend to others.
39
+
40
+
## Simulators and ROS
41
+
42
+
RL-Studio supports [ROS](http://wiki.ros.org/) Noetic which is necesary to interact with [Gazebo](https://classic.gazebosim.org/) or also as a bridge with [Carla](https://carla.readthedocs.io/en/0.9.13/). Although Carla can work without ROS also. Currently there are working canonical reinforcement lerning tasks with OpenAI simulator through [gymnasium](https://gymnasium.farama.org/) library.
43
+
44
+
# Installation
45
+
46
+
## Install ROS
47
+
48
+
RL-Studio works with ROS Noetic. You can [install ROS Noetic from the official documentation](http://wiki.ros.org/noetic/Installation/Ubuntu) and installing ROS Noetic Full Desktop.
_It is highly recommended to create a virtual environment:_
41
66
42
67
```bash
43
-
poetry install
68
+
cd RL-Studio
69
+
pip install -r requirements.txt
44
70
```
45
71
46
-
### Install dependencies using pip (not recommended):
47
-
48
-
_Note: In case you don't want to use Poetry as a dependency manager, you can install it with pip as follows (previously it is highly recommended to create a virtual environment):_
There will be as many variables as there are circuits to be executed. In case you want to work with other circuits or agents, there will be necessary add the correct paths to variables in `.bashrc` file in the same way.
121
155
156
+
And finally, do not forget adding
157
+
```bash
158
+
export PYTHONPATH=$PYTHONPATH:PATH/TO/RL-Studio
159
+
```
160
+
161
+
## Usage/Examples
162
+
122
163
To check that everything is working correctly you can try launching a ROS exercise by typing:
cd /PATH/TO/RL-Studio/rl_studio/CustomRobots/f1/launch
126
167
roslaunch simple_circuit.launch
127
168
```
128
169
129
-
And to begin training and inferencing, please go to [README.md](https://github.com/JdeRobot/RL-Studio/blob/main/rl_studio/README.md)
170
+
and you could see something similar to the screenshot
171
+
172
+
173
+
174
+
175
+
# Work with RL-Studio
176
+
177
+
178
+
Additional information on how to create, run and test reinforcement learning models, how to create a configuration file to launch the application and to begin training and inferencing, please go to [rl-studio](https://github.com/JdeRobot/RL-Studio/blob/main/rl_studio/README.md).
179
+
180
+
Information about coding or naming classes and files, how the directory structure is designed and where to save models, metrics, logs and graphics, please go to [codig style file](https://github.com/JdeRobot/RL-Studio/blob/main/CODING.md).
181
+
182
+
FAQ please go to [answering questions](https://github.com/JdeRobot/RL-Studio/blob/main/FAQ.md).
183
+
184
+
# Reference
185
+
186
+
A paper about RL-Studio appears in Volume **590** of the **Lecture Notes in Networks and Systems** series of Springer and can be cited with bibtex entry:
187
+
188
+
```
189
+
@inproceedings{fernandez2023rl,
190
+
title={RL-Studio: A Tool for Reinforcement Learning Methods in Robotics},
191
+
author={Fern{\'a}ndez de Cabo, Pedro and Lucas, Rub{\'e}n and Arranz, Ignacio and Paniego, Sergio and Ca{\~n}as, Jos{\'e} M},
192
+
booktitle={Iberian Robotics conference},
193
+
pages={502--513},
194
+
year={2023},
195
+
organization={Springer}
196
+
}
197
+
```
198
+
or
199
+
```text
200
+
Fernández de Cabo, P., Lucas, R., Arranz, I., Paniego, S., & Cañas, J. M. (2023). RL-Studio: A Tool for Reinforcement Learning Methods in Robotics. In Iberian Robotics conference (pp. 502-513). Springer, Cham.
201
+
```
202
+
# Contributing
203
+
204
+
Contributions are always welcome!
205
+
206
+
See [CONTRIBUTING](CONTRIBUTING.md) for ways to get started.
207
+
208
+
Please adhere to this project's `code of conduct`.
0 commit comments