For a control system of 4 states and 2 control inputs...

[CanChen-Caesar]

Hi Simon,

I am trying to implement it to a four-state two-input continuous-time autonomous system. May I know what modifications should be made? Thanks a lot in advance.

Caesar

[SimonRennotte]

Hi,

1. Look at the list of limitations of the method and check that it will still work for your system.
   Specifically, if your system doesn't have too much noise and doesn't have a delay between actions and their effect on observations

2. You must either create a gym environment like here: https://www.novatec-gmbh.de/en/blog/creating-a-gym-environment/,
   or if you prefer, create a python object with the same attributes and functions as a gym env, and call it instead in the main script.

3. Copy main_parameters_pendulum.json and rename it main_parameters_"myenv".json. Set
   - env_to_control to the name of your gym env (if you use a gym env)
   - render_env to false if you did not provide the render method for your env/object
   - num_steps_env to a sufficently large number to observe the results of your test
   - random_actions_init to the number of random actions you want at the beginning before the usage of the control object

4. Copy parameters_pendulum.json and rename it parameters_"myenv".json. Set
   - noise_covar.noise to a list with whose len is the number of state you have
   - base_kernel.lengthscale to a list whose len is the number of state you have (or number of state x (number of state +
   number of inputs))
   - outputscale to a list with whose len is the number of state you have
   - target_state_norm to the value you want for each of the normalized state (normalized with the max and min value
   provided in action_space and observation space of the gym env object)
   - weight_state to the weight you attribute each state in the cost function
   - weight_state_terminal to the weight you attribute to each state at the end of the prediction horizon
   - target_action_norm to the value you want for each of your normalized inputs. For example 0.5 means no torque for the
   pendulum case
   - weight_action to the weight you attribute to action target for the cost
   - in general other parameters that have a list of 3 items for the pendulum case to a list of 4 items, 1 for each state
   - in general other parameters that have a list of 1 item to a list of 2 items, 1 for each actions

5. replace the path in the first line of the main.py script to your main parameter file

For specific information about tuning the parameters, specific knowledge about the system is needed. You can email me with details if you want more specific information.

Let me know if it worked.
Also, if you agree, please send me or post the results of the graphs here, I'm interested to see the results for other systems.

Thank you.

[CanChen-Caesar]

Thanks very much for your reply. I will let you know whether it works. Hopefully, I will get back to you in a few days.

[SimonRennotte]

Any news? If it is too complicated, I can change the code to simplify usage for other environments. What steps did block you or did you have difficulty with?

[CanChen-Caesar]

Hi, Simon. Thanks very much for your attention. I just got some results about 20 hours ago.

First, this is the four-state two-input continuous-time system I want to implement the GP-RL-MPC method to.

where the state is [n_11, n_12, n_21, n_22] and the input is [u_12, u_21]. [q_11, q_12, q_21, q_22] is constant. Note that I define M_ij (t) as n_ij (t) / n_i (t) with i, j = 1, 2. If you want more information of this system, you may refer to this paper

Geroliminis, N., Haddad, J., & Ramezani, M. (2012). Optimal perimeter control for two urban regions with macroscopic fundamental diagrams: A model predictive approach. IEEE Transactions on Intelligent Transportation Systems, 14(1), 348-359.

via this link https://infoscience.epfl.ch/record/182816/files/NG_JH_MR_2012.pdf . However, different from this paper, my control objective is to stabilize the states to a predefined steady state.

I still have difficulty with installing my own gym environment. Here is the error I got.

Could you please let me know what could be the problem?

After a few days of trials, I decided to do something bad. I copied the code in my own environment to Pendulum-v0 and finally got it run without errors. Here are the simulations.

Finally, I think it works! And I must put effort into tuning the parameters from now on.

[SimonRennotte]

Thank you for the precisions.
For ease of use, I have pushed an example of gym env creation in the folder utils, so you don't have to do the complex integration.
I have also fixed a bug that occured for envs with multi-dimensional actions.
It should work now.
But it seems from your graph that your states are constant. It will not work if it is the case.
Are they constant or the variations are just very small?
It could be an implementation error if they are constant.

[CanChen-Caesar]

Thank you for your input!
I think the variations are just too small but I will double-check it.