Applying the Reinformer architecture to minimal RNNs

[WhyAreYouJay]

Hi there,
I have a repository, where we have been trying to get the Reinformer to work with a minimal Recurrent Network.

However, both on Hopper and medium-expert datasets we just cannot reach anywhere near the levels of the reinformer (or the original minimal RNNs for that matter)

I was wondering if you might have an idea as to what could be wrong? I have implemented dropout in the minRNN similar to what you have done for MultiHeadAttention, with some slight improvements, but we are still a ways from the results I expected based on the Reinformer and MinRNN baselines.

If you are curious, the repo can be found here

Either way, cool repo!

[Dragon-Zhuang]

@WhyAreYouJay Thank you very much for your recognition of our work!

I would like to know which environments you are unable to achieve the original RNN results on? Is it all the environments of hopper and medium-expert, or just one hopper-medium-expert environment?

Additionally, when the return-to-go is input into the network, is there any normalization, since its scale varies greatly compared to the ranges of state and action? This is an issue that does not exist in the original DT or RNN. The code for reward normalization can be found in d4rl_trajectory_dataset.py (Line33~43).