Calculating rewards for multyagent task

[Ange1ika]

Hello! I want to learn how to control a swarm of UAVs with obstacles. Could you tell me, please, in what order to change the code? Which action to use and how to work with it correctly. How to figure out the rgb calculation?

I'm figuring out the pipeline, I found adding obstacles, but there are difficulties with calculating rewards. Can you explain how you got these threshold values? Why such a formula

#### Target cumulative rewards (problem-dependent) ########## if DEFAULT_ACT == ActionType.ONE_D_RPM: target_reward = 474.15 if not multiagent else 949.5 else: target_reward = 467. if not multiagent else 920. callback_on_best = StopTrainingOnRewardThreshold(reward_threshold=target_reward, verbose=1)

[Ange1ika]

During training I encountered a problem - it only works in height and is limited in the rest of the plane....
I set such target:
self.TARGET_POS = self.INIT_XYZS + np.array([[-5/(i+1),0.5, 0.5] for i in range(num_drones)])
And I changed thresh's in MultiHoverAviary
` def _computeTruncated(self):
"""Computes the current truncated value.

    Returns
    -------
    bool
        Whether the current episode timed out.

    """
    states = np.array([self._getDroneStateVector(i) for i in range(self.NUM_DRONES)])
    for i in range(self.NUM_DRONES):
        if (abs(states[i][0]) > 2.0 or abs(states[i][1]) > 2.0 or states[i][2] > 2.0 # Truncate when a drones is too far away
         or abs(states[i][7]) > .4 or abs(states[i][8]) > .4 # Truncate when a drone is too tilted
        ):
            return True
    if self.step_counter/self.PYB_FREQ > self.EPISODE_LEN_SEC:
        return True
    else:
        return False`

[JacopoPan]

Hi @Ange1ika

a few comments:

• to create your problem-specific learning environment, you need to subclass BaseRLAviary
• MultiHoverAviary is one such example of a BaseRLAviary subclass
• The reward function is then computed in this subclass by method _computeReward()
• The ActionTypes that are prefixed by ONE_D_ only allow movement on the z-axis, you will want to use PID or VEL

[Ange1ika]

Hi @Ange1ika

a few comments:

• to create your problem-specific learning environment, you need to subclass BaseRLAviary
• MultiHoverAviary is one such example of a BaseRLAviary subclass
• The reward function is then computed in this subclass by method _computeReward()
• The ActionTypes that are prefixed by ONE_D_ only allow movement on the z-axis, you will want to use PID or VEL

Thank you! I chose pid control now.

Am I correct in understanding that multi-agent learning requires using RGB space (taking obstacles into account). Then, the distance to objects and reward information can be obtained via RGB? Can I see some examples? Here, if I've understood correctly, it's only without obstacles:

gym-pybullet-drones/experiments/learning
/multiagent.py

[JacopoPan]

I’d definitely recommend expanding the KIN observation space to account for drone positions, using RBG with multiple vehicles would be computationally very wasteful.

[Ange1ika]

I’d definitely recommend expanding the KIN observation space to account for drone positions, using RBG with multiple vehicles would be computationally very wasteful.

But with kin I won't be able to add obstacles or will I?

[Ange1ika]

I'm currently using MLP, but the drones freeze at times. I need more time; RGB observations are better handled by CNN.

And I'm still thinking about what policy to follow; it would be good to have different actors and one critic, as in your article.

[JacopoPan]

I’d definitely recommend expanding the KIN observation space to account for drone positions, using RBG with multiple vehicles would be computationally very wasteful.

But with kin I won't be able to add obstacles or will I?

You could simply extend the observation with position and dimension of the obstacles

[Ange1ika]

I’d definitely recommend expanding the KIN observation space to account for drone positions, using RBG with multiple vehicles would be computationally very wasteful.

But with kin I won't be able to add obstacles or will I?

You could simply extend the observation with position and dimension of the obstacles

Okey, I understood. Thank you very much!!