Error in NERSC's Perlmutter

[mihelog]

Hello everyone!
I've been trying to run EAMxx. The script to build and execute it works, but I get a weird runtime error. I wonder if this is related to some recent Perlmutter software changes but I'm a recent user so I have no previous data points. Thanks for advance!

The error shows up in the run log, filename: pm-gpu/se09-aug20_take1/t.ne30pg2_ne30pg2.F2010-SCREAMv1.se09-aug20_take1.5d.n0001t4x111XX1.nocosp/run/e3sm.log.36558505.250305-171531

The error:

0: CalcWorkPerBlock: Total blocks:         72 Ice blocks:         72 IceFree blocks:          0 Land blocks:          0
0: MCT::m_Router::initp_: GSMap indices not increasing...Will correct
0: MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
0: MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
0: MCT::m_Router::initp_: GSMap indices not increasing...Will correct
0: bfbhash>              0 a9a787e7e84eb9b1 (Hommexx)
0:
0:  FAIL:
0: f.time_dim==nullptr
0: /global/cfs/cdirs/m888/ThrustE/mihelog/E3SM/components/eamxx/src/share/io/eamxx_scorpio_interface.cpp:1093
0: Error! Attempt to redeclare unlimited dimension.
0:  - filename: scream_output.3hourlyAVG.h.AVERAGE.nhours_x3.2022-10-04-00000.nc
0:
1:
1:  FAIL:
1: f.time_dim==nullptr
1: /global/cfs/cdirs/m888/ThrustE/mihelog/E3SM/components/eamxx/src/share/io/eamxx_scorpio_interface.cpp:1093
1: Error! Attempt to redeclare unlimited dimension.
1:  - filename: scream_output.3hourlyAVG.h.AVERAGE.nhours_x3.2022-10-04-00000.nc
1:
2:
2:  FAIL:
2: f.time_dim==nullptr
2: /global/cfs/cdirs/m888/ThrustE/mihelog/E3SM/components/eamxx/src/share/io/eamxx_scorpio_interface.cpp:1093
2: Error! Attempt to redeclare unlimited dimension.
2:  - filename: scream_output.3hourlyAVG.h.AVERAGE.nhours_x3.2022-10-04-00000.nc
2:
3:
3:  FAIL:
3: f.time_dim==nullptr
3: /global/cfs/cdirs/m888/ThrustE/mihelog/E3SM/components/eamxx/src/share/io/eamxx_scorpio_interface.cpp:1093
3: Error! Attempt to redeclare unlimited dimension.
3:  - filename: scream_output.3hourlyAVG.h.AVERAGE.nhours_x3.2022-10-04-00000.nc
3:
0: Used 0 of out of 1048576049, high_water was 513692248
2: Used 0 of out of 1048576049, high_water was 409929968
3: Used 0 of out of 1048576049, high_water was 518303848
1: Used 0 of out of 1048576049, high_water was 518303848
0: terminate called after throwing an instance of 'std::runtime_error'
0:   what():
0:  FAIL:
0: it.second.num_customers==0
0: /global/cfs/cdirs/m888/ThrustE/mihelog/E3SM/components/eamxx/src/share/io/eamxx_scorpio_interface.cpp:366
0: Error! ScorpioSession::finalize called, but a file is still in use elsewhere.
0:  - filename: scream_output.3hourlyAVG.h.AVERAGE.nhours_x3.2022-10-04-00000.nc
2: terminate called after throwing an instance of 'std::runtime_error'
2:   what():
2:  FAIL:
2: it.second.num_customers==0
2: /global/cfs/cdirs/m888/ThrustE/mihelog/E3SM/components/eamxx/src/share/io/eamxx_scorpio_interface.cpp:366
2: Error! ScorpioSession::finalize called, but a file is still in use elsewhere.
2:  - filename: scream_output.3hourlyAVG.h.AVERAGE.nhours_x3.2022-10-04-00000.nc
2:
0:
0: Program received signal SIGABRT: Process abort signal.
0:
0: Backtrace for this error:
2:
2: Program received signal SIGABRT: Process abort signal.
2:
2: Backtrace for this error:
1: terminate called after throwing an instance of 'std::runtime_error'
1:   what():
1:  FAIL:
1: it.second.num_customers==0
1: /global/cfs/cdirs/m888/ThrustE/mihelog/E3SM/components/eamxx/src/share/io/eamxx_scorpio_interface.cpp:366
1: Error! ScorpioSession::finalize called, but a file is still in use elsewhere.
1:  - filename: scream_output.3hourlyAVG.h.AVERAGE.nhours_x3.2022-10-04-00000.nc
1:
1:
1: Program received signal SIGABRT: Process abort signal.
1:
1: Backtrace for this error:
3: terminate called after throwing an instance of 'std::runtime_error'
3:   what():
3:  FAIL:
3: it.second.num_customers==0
3: /global/cfs/cdirs/m888/ThrustE/mihelog/E3SM/components/eamxx/src/share/io/eamxx_scorpio_interface.cpp:366
3: Error! ScorpioSession::finalize called, but a file is still in use elsewhere.
3:  - filename: scream_output.3hourlyAVG.h.AVERAGE.nhours_x3.2022-10-04-00000.nc
3:
3:
3: Program received signal SIGABRT: Process abort signal.

[bartgol]

This often happens when you have 2+ output stream trying to write on the same file. Which, in turns, happens if in 2+ of your yaml files you specify the same prefix, average type, frequency, AND frequency units.

[mihelog]

Thank you this seems to be it! At least no error is generated any more. Much appreciated.