Hard to debug IndexError

[Sebastian-Griesbach]

Hey People,
I'm encountering an error sporadically which unfortunately I don't know how to reproduce and debug. The issue is most likely with my code but the error comes from the Replay buffer but I am not sure what it means.
There error I'm receiving is the following:

IndexError: index 1000000 is out of bounds for axis 0 with size 1000000
Exception ignored in: 'cpprb.PyReplayBuffer.ReplayBuffer.add_cache_i'
Traceback (most recent call last):
File "/home/.../buffer.py", line 198, in _add_episodic_data
self.replay_buffer.add(**episodic_data_dictionaries[-1])
IndexError: index 1000000 is out of bounds for axis 0 with size 1000000

The replay buffer is set up to use next_of and stack_compression for atari environments.
In my code I build a wrapper around the replay buffer such that it calls .on_episode_end() correctly without relying on being handed full episodes on each .add() call. My function for that looks like this:

def _add_episodic_data(self, transition_dictionary: Dict[str, np.ndarray], episode_ended: np.ndarray) -> None:
        #transition_dictionary is a transition dictionary over an arbitrary amount of transitions possibly from multiple episodes.
        #episode_ended is a numpy boolean array of the length of number transition which indicates whether a specific transition was terminal or not
        batch_size = len(transition_dictionary[self.some_data_key])

        episode_start_ids = np.where(episode_ended)[0] + 1
        transition_data_array_list = list(transition_dictionary.values())

        # Split Transition dictionary into a list of dictionaries such that each dictionary corresponds to an episode
        episodic_data_list = map(lambda transition_data_array: np.split(
            transition_data_array, episode_start_ids, axis=0), transition_data_array_list)
        episodic_data_dictionaries = list(map(lambda episodic_data: dict(
            zip(transition_dictionary.keys(), episodic_data)), zip(*episodic_data_list)))

        # Add data with an ending episodes
        for i in range(len(episode_start_ids)):
            self.replay_buffer.add(**episodic_data_dictionaries[i])
            self.replay_buffer.on_episode_end()

        # Add remaining data without ending episode
        if episode_start_ids.size < 1 or episode_start_ids[-1] < batch_size:
            self.replay_buffer.add(**episodic_data_dictionaries[-1]) # This line triggers the error (very occasionally)

The problem might lie in there or not. But you don't need to worry about my code much, what I would like to know is what the error message given at the top could be indicative of. So that I might know what to look for to debug this or find a way to reproduce the error reliably. For example, this error means that I pass an empty dictionary to the buffer or something like that.

I would be very happy if someone had an Idea what this index out-of-bound error could mean.

Btw this library is immensely useful to me as I'm currently building my own Reinforcement Learning Framework for my research. Thank you for making this thing.

[ymd-h]

Hi @Sebastian-Griesbach
Thank you for your feedback.

Can you give us all the arguments you passed to the constructor of the replay buffer?

import cpprb

rb = cpprb.ReplayBuffer(
 # here
)

Out of range might be a bug.
We will investigate.

[Sebastian-Griesbach]

Of course, in my code, I initialize the replay buffer as follows:

        self.replay_buffer = cpprb.ReplayBuffer(
            max_size, replay_buffer_info, **additional_arguments)

With the following values (This is the output I get when I print the replay_buffer_info, the dictionary itself is created dynamically based on the environment and some other factors):

max_size = 1000000
replay_buffer_info = {'rewards': {'shape': (1,), 'dtype': <class 'numpy.float32'>}, 'observations': {'shape': (84, 84, 4), 'dtype': 'uint8'}, 'next_observations': {'shape': (84, 84, 4), 'dtype': 'uint8'}, 'actions': {'shape': (1,), 'dtype': 'int64'}, 'terminated': {'shape': (1,), 'dtype': <class 'numpy.bool_'>}}
additional_arguments = {'next_of': 'observations', 'stack_compress': ['observations', 'next_observations']}

Btw my DQN implementation with this seems to work fine. So the error does not seem to be breaking.

[Sebastian-Griesbach]

I spent a bit more time and found a minimal code example to reproduce the error:

import cpprb
import numpy as np

max_size = 10
replay_buffer_info = {'rewards': {'shape': (1,), 'dtype': np.float32}, 'observations': {'shape': (84, 84, 4), 'dtype': 'uint8'}, 'next_observations': {'shape': (84, 84, 4), 'dtype': 'uint8'}, 'actions': {'shape': (1,), 'dtype': 'int64'}, 'terminated': {'shape': (1,), 'dtype': np.bool_}}
additional_arguments = {'next_of': 'observations', 'stack_compress': ['observations', 'next_observations']}

buffer = cpprb.ReplayBuffer(max_size, replay_buffer_info, **additional_arguments)

dummy_data = {'rewards': np.array([1.]), 'observations': np.array([np.ones((84, 84, 4), dtype=np.uint8)]), 'next_observations': np.array([np.ones((84, 84, 4), dtype=np.uint8)]), 'actions': np.array([1]), 'terminated': np.array([False])}

for i in range(8):
    buffer.add(**dummy_data)

buffer.on_episode_end()

for i in range(8):
    buffer.add(**dummy_data)

The error occurs exactly at the 16th iteration of the adding loop. regardless of if you call .on_episode_end() in between or not. I hope this helps with the investigation.

[ymd-h]

@Sebastian-Griesbach

Thank you for your effort.
I confirm there is a bug when we use next_of and memory_compress and will fix as soon as possible.

FYI:

Although there is a bug, you also did a mistake.
When you use next_of feature, you cannot add next_XXX to env_dict (and stack_compress) manually.
Please see section '1.2' and '2.2' at https://ymd_h.gitlab.io/cpprb/features/memory_compression/

[Sebastian-Griesbach]

Happy to help you help me. :D
Thank you for making we aware of my configuration mistake!

Could you write a final message here once the bug is fixed?

[ymd-h]

@Sebastian-Griesbach

Could you write a final message here once the bug is fixed?

Of course.
I will write message here after release.

[ymd-h]

@Sebastian-Griesbach

We released cpprb v10.7.2 with the fix for this bug.
Please try it.

Thank you for your help.

[Sebastian-Griesbach]

Seems to be fixed. Thanks alot!