Reproduce Query

[mhngu23]

Dear, authors!
I appreciate your paper and your code. I am currently trying to reproduce the experimental result.
However, I have a few queries regarding the results for Maze2d environments, specifically for medium and large environments. Although I can reproduce the results on all other environments, I cannot seem to get the same results for these two environments that I have mentioned. I wonder whether this is a problem with torch or gym version, so can you please help providing the version that you have used to run your code?

Thank you so much for your time!
Best Regards

[JoshuaWenHIT]

I also encountered the same problem as the questioner. Below is my error report. Strangely, this problem only occurs in the medium and large environments of maze2d.

Traceback (most recent call last):
File "/home/lab-1523/JoshuaWen/Code/Reinformer/main.py", line 207, in
experiment(vars(args))
File "/home/lab-1523/JoshuaWen/Code/Reinformer/main.py", line 126, in experiment
loss = Trainer.train_step(
File "/home/lab-1523/JoshuaWen/Code/Reinformer/trainer.py", line 85, in train_step
) = self.model.forward(
File "/home/lab-1523/JoshuaWen/Code/Reinformer/model/reinformer.py", line 115, in forward
action_dist_preds = self.predict_action(h[:, 1]) # predict action given s, R
File "/home/lab-1523/anaconda3/envs/ae-rvs/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lab-1523/JoshuaWen/Code/Reinformer/model/net.py", line 146, in forward
return pyd.Normal(mu, std)
File "/home/lab-1523/anaconda3/envs/ae-rvs/lib/python3.9/site-packages/torch/distributions/normal.py", line 56, in init
super().init(batch_shape, validate_args=validate_args)
File "/home/lab-1523/anaconda3/envs/ae-rvs/lib/python3.9/site-packages/torch/distributions/distribution.py", line 62, in init
raise ValueError(
ValueError: Expected parameter loc (Tensor of shape (128, 10, 2)) of distribution Normal(loc: torch.Size([128, 10, 2]), scale: torch.Size([128, 10, 2])) to satisfy the constraint Real(), but found invalid values:
tensor([[[nan, nan],
[nan, nan],
[nan, nan],
...,
[nan, nan],
[nan, nan],
[nan, nan]],

    [[nan, nan],
     [nan, nan],
     [nan, nan],
     ...,
     [nan, nan],
     [nan, nan],
     [nan, nan]],

    [[nan, nan],
     [nan, nan],
     [nan, nan],
     ...,
     [nan, nan],
     [nan, nan],
     [nan, nan]],

    ...,

    [[nan, nan],
     [nan, nan],
     [nan, nan],
     ...,
     [nan, nan],
     [nan, nan],
     [nan, nan]],

    [[nan, nan],
     [nan, nan],
     [nan, nan],
     ...,
     [nan, nan],
     [nan, nan],
     [nan, nan]],

    [[nan, nan],
     [nan, nan],
     [nan, nan],
     ...,
     [nan, nan],
     [nan, nan],
     [nan, nan]]], device='cuda:0', grad_fn=<TanhBackward0>)

[Dragon-Zhuang]

@mhngu23 @JoshuaWenHIT Thank you for your attention to my work!

Regarding this issue, the key lies in the data processing part download_d4rl_datasets.py. Now the Line 39 is
dataset = env.get_dataset()
while the first version is
dataset = d4rl.qlearning_dataset(env).

We made this modification because the next_observation in the q_learning dataset is unnecessary, so we opted for a simplified version. Then we found that the maze2d environment exhibited NaN values, which we had not observed in our previous experiments. We are not sure about the cause of this issue, as the datasets obtained from both methods should be the same!

If you wish to reproduce the results from the original paper, it is recommended to use the first version of the code. We are also in the process of troubleshooting this bug. If you have any insights, please feel free to share and discuss!