reproducing PEMS08

[naivegillian]

Hi, I am trying to reproduce the results. METR-LA and PEMS-BAY looks fine.
But on PEMS08 the results is not good as expected.
Is the adjacency matrix used for PEMS08 can be downloaded from
https://drive.google.com/drive/folders/1H3nl0eRCVl5jszHPesIPoPu1ODhFMSub
so the first 5 nodes will be
0. 0. 0. 0. 0
0. 0. 1. 0. 0
0. 1. 0. 0. 1
0. 0. 0. 0. 0
0. 0. 1. 0. 0
Also, I find there are more 'inh_layer.pos_encoder' in my trained model than the one on
https://drive.google.com/drive/folders/18nkluGajYET2F9mxz3Kl6jcFVAAUGfpc
Thanks

[huangst21]

Hi, I am trying to reproduce the results. METR-LA and PEMS-BAY looks fine. But on PEMS08 the results is not good as expected. Is the adjacency matrix used for PEMS08 can be downloaded from https://drive.google.com/drive/folders/1H3nl0eRCVl5jszHPesIPoPu1ODhFMSub so the first 5 nodes will be 0. 0. 0. 0. 0 0. 0. 1. 0. 0 0. 1. 0. 0. 1 0. 0. 0. 0. 0 0. 0. 1. 0. 0 Also, I find there are more 'inh_layer.pos_encoder' in my trained model than the one on https://drive.google.com/drive/folders/18nkluGajYET2F9mxz3Kl6jcFVAAUGfpc Thanks

I had the same problem with the PEMS08 dataset, the experimental results were a long way from the reported results.

[zezhishao]

Hi guys.
Thank you for your interest in D2STGNN.
This repository is currently not under maintenance.
You can reproduce D2STGNN based on BasicTS, which ensures the fairness of results based on a unified training/evaluation pipeline. You can also find more baselines and datasets in BasicTS.
Here are the results of D2STGNN in BasicTS:

2023-08-31 12:09:26,902 - easytorch-training - INFO - Epoch 100 / 100
2023-08-31 12:11:16,473 - easytorch-training - INFO - Result <train>: [train_time: 109.57 (s), lr: 6.25e-05, train_MAE: 13.613422, train_RMSE: 22.730559, train_MAPE: 0.089726, train_WAPE: 0.059244, train_MSE: 519.687916]
2023-08-31 12:11:16,475 - easytorch-training - INFO - Start validation.
2023-08-31 12:11:26,387 - easytorch-training - INFO - Result <val>: [val_time: 9.91 (s), val_MAE: 14.225937, val_RMSE: 23.114646, val_MAPE: 0.111527, val_WAPE: 0.066082, val_MSE: 592.654739]
2023-08-31 12:11:26,438 - easytorch-training - INFO - Checkpoint checkpoints/D2STGNN_100/8407cac278902dac451f6fd1891a2471/D2STGNN_best_val_MAE.pt saved
2023-08-31 12:11:35,333 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 12.352158, Test RMSE: 19.550905, Test MAPE: 0.081585, Test WAPE: 0.053134, Test MSE: 382.237885
2023-08-31 12:11:35,364 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 12.845043, Test RMSE: 20.666191, Test MAPE: 0.085214, Test WAPE: 0.055268, Test MSE: 427.091431
2023-08-31 12:11:35,394 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 13.215744, Test RMSE: 21.467710, Test MAPE: 0.088085, Test WAPE: 0.056878, Test MSE: 460.862610
2023-08-31 12:11:35,424 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 13.593189, Test RMSE: 22.360163, Test MAPE: 0.089042, Test WAPE: 0.058517, Test MSE: 499.976868
2023-08-31 12:11:35,461 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 13.862035, Test RMSE: 22.866516, Test MAPE: 0.091356, Test WAPE: 0.059690, Test MSE: 522.877563
2023-08-31 12:11:35,485 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 14.095591, Test RMSE: 23.339945, Test MAPE: 0.093465, Test WAPE: 0.060712, Test MSE: 544.752991
2023-08-31 12:11:35,509 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 14.365138, Test RMSE: 23.942684, Test MAPE: 0.093750, Test WAPE: 0.061890, Test MSE: 573.252136
2023-08-31 12:11:35,533 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 14.566394, Test RMSE: 24.324722, Test MAPE: 0.095804, Test WAPE: 0.062775, Test MSE: 591.692139
2023-08-31 12:11:35,556 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 14.769111, Test RMSE: 24.696266, Test MAPE: 0.097954, Test WAPE: 0.063666, Test MSE: 609.905579
2023-08-31 12:11:35,580 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 14.968579, Test RMSE: 25.066189, Test MAPE: 0.098992, Test WAPE: 0.064545, Test MSE: 628.313782
2023-08-31 12:11:35,603 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 15.147799, Test RMSE: 25.366919, Test MAPE: 0.101015, Test WAPE: 0.065337, Test MSE: 643.480591
2023-08-31 12:11:35,627 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 15.400373, Test RMSE: 25.736530, Test MAPE: 0.103332, Test WAPE: 0.066446, Test MSE: 662.368958
2023-08-31 12:11:36,137 - easytorch-training - INFO - Result <test>: [test_time: 9.70 (s), test_MAE: 14.098426, test_RMSE: 23.357384, test_MAPE: 0.093299, test_WAPE: 0.060734, test_MSE: 545.567383]
2023-08-31 12:11:36,194 - easytorch-training - INFO - Checkpoint checkpoints/D2STGNN_100/8407cac278902dac451f6fd1891a2471/D2STGNN_100.pt saved
2023-08-31 12:11:36,201 - easytorch-training - INFO - The training finished at 2023-08-31 12:11:36

Kindly note that considering the randomness, we can not always get extactly the same results as reported in the paper.
Reproduced results might be slightly different, better or worse.

[naivegillian]

Is there any difference in settings such that you can finish in 100 epoches ?
I see 'standard_transform' in basicts. Thank you.