Error when evaluating transfuser on H100

[ChengenXie]

Hi, authors. Thanks for your great work for the community!
I've trained transfuser with the official code on H100 and there is no error when evaluating it on H100. However, I got 0of PDM Score. This result is very strange, so I retest it on 4090 and get reasonable results.
Have you encountered a similar issue, or do you have any thoughts about this problem?
Thank you in advance!

[yugao2011]

I experience the same issue when evaluate on H20 with Gen4 Xeon Scalable CPU.

[EllingtonKirby]

I can report the same issue with several different agents on H100 gpus. Specifically the 0.0 EPDMS comes from a 0.0 score on the drivable area compliance, which is then multiplied against the other values.

When using a V100 I have reasonable scores.

To confirm the issue I tried to control some randomness on my machines via:

import torch, random
SEED=42
random.seed(SEED); np.random.seed(SEED)
torch.manual_seed(SEED); torch.cuda.manual_seed_all(SEED)
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.deterministic=True
torch.backends.cudnn.benchmark=False
torch.backends.cuda.matmul.allow_tf32=False
torch.backends.cudnn.allow_tf32=False
try:
    torch.backends.cuda.sdp_kernel(enable_flash=False, enable_math=True, enable_mem_efficient=False)
except: pass

But this did not change anything.

For reference I am working on a cluster where using an H100 requires torch==2.1.0 minimum and certain other dependencies via loading a slurm module (not sure exactly what is in there yet...)

[DanielDauner]

This is somewhat strange. The torch version and GPU hardware should only affect the model inference but not the metric itself. Could this issue be related to #28 or are there any updates here?

Best,
Daniel