Consider changing derecho testing to use the shared develop queue

[billsacks]

Our derecho testing currently uses dedicated 128 core nodes for each test run. This is overkill for our needs, where 8 cores should be sufficient.

I tried changing to use the shared 8-core develop nodes with these diffs:

diff --git a/config/derecho.yaml b/config/derecho.yaml
index 7fc0cf2..038533c 100644
--- a/config/derecho.yaml
+++ b/config/derecho.yaml
@@ -1,11 +1,12 @@
 machine:
   name: derecho
- cores_per_node: 128
+ # Use just 8 cores on the develop queue for less resource wastage (the develop queue allows shared nodes) and potentially less queue wait time
+ cores_per_node: 8
  head_node_name: derecho6
  scheduler:
  type: pbs
  account: p93300606
- queue: main
+ queue: develop
  partition: None
 
 test:

This led to a few test failures:

gfortran-pio2.5.10-O: failure in ESMF_ArrayRedistPerfUTest.F90: slightly exceeds tolerance (1.64e-2 > 1e-2)

gfortran-pio2.5.10-g:

CRASHED: mpi/g: src/Superstructure/Component/tests/ESMF_CompTunnelUTest.F90
   FAIL: mpi/g:   Check time delay on timeout ServiceLoop Actual Component E, ESMF_CompTunnelUTest.F90, line 1381:  Incorrect time delay
   FAIL: mpi/g:   ServiceLoop for the Actual Component E - timeoutFlag, ESMF_CompTunnelUTest.F90, line 1407:  Did not return ESMF_SUCCESS
   FAIL: mpi/g:   Testing timeoutFlag, ESMF_CompTunnelUTest.F90, line 1414:  timeoutFlag wrong

In addition, there were a number of other issues with failures to report / commit to artifacts repo, but at least some of these may have been due to issues with the derecho qstat command (see #104).

I'd like to try this again to see if we can get testing to work reliably from the develop queue, since this should greatly decrease the cost incurred with our nightly tests.

[theurich]

Does this mean that multiple ESMF test jobs might run on the same Derecho node concurrently? I am asking because this will likely be an issue with ESMF_CompTunnelUTest.F90.

[billsacks]

Hmmm, thanks @theurich - yes, this does mean that multiple ESMF test jobs almost certainly will run on the same Derecho node concurrently. So I guess we'd either need to abandon this test on this system (which doesn't seem ideal since Derecho is one of our primary test platforms), abandon this plan, or come up with some other alternative. Maybe testing on the smaller casper cluster (described in https://arc.ucar.edu/docs... I can't manage to get a link to the exact place, but you can search for casper there), or izumi. Or just let things go as is for now. I feel like, long-term, it could be good to reduce our unnecessarily large usage... this ESMF testing uses a substantial fraction of the total CESM Software Engineering Working Group allocation... but so far it still seems to be okay.... Maybe we can talk briefly about this at the core team meeting tomorrow.

[billsacks]

From discussion at today's core team meeting:

There is general support for reducing our resource usage if possible.

We could consider a compromise like:

1. Run most derecho tests on the shared develop nodes, but one combo on dedicated nodes.
2. Or maybe better: Run nightly derecho tests all on shared develop nodes; these would skip some tests. But have something like weekly tests that run on dedicated nodes; these would run all tests. This could be managed by having two separate configs, one for derecho-shared and one for derecho-dedicated; and then have separate cron jobs controlling each.

[billsacks]

I sent a help ticket to the UCAR help desk asking if they have any other advice / ideas for how we might best accomplish this.

One suggestion was:

Have a look at "Running multiple MPI applications in a single job" here:
https://ncar-hpc-docs.readthedocs.io/en/latest/compute-systems/derecho/starting-derecho-jobs/derecho-job-script-examples/

This approach takes a dedicated 128 node and runs N jobs each requiring M processors simultaneously; provided of course M*N<=128.

So you'd request a 128 node, but it would be responsible for <=16 of your tests at a time; since you only need 8 per test.

If your jobs for similar lengths; this would make the most sense. You send them off in groups of <=16 to a single node.

If a couple finish quickly; no problem. But if you have one 'tail' job that runs much much longer; there is still resource waste.

This introduces the complication of aggregaring your tests; but let me know if you think that is doable. It is probably the easiest way to go. If this is not workable for some reason I've got other ideas we can explore; that would still keep you in the main queue on Derecho on dedicated nodes.

However, I don't think that will help us, at least with the CompTunnel tests and maybe also with the performance-related tests, since the problem we're encountering is related to contention between our own test jobs.

It sounds like they may have some other suggestions; I'll post back if anything seems helpful.

[billsacks]

I reran the full derecho testing using 8 cores per combo on the shared develop queue, with these diffs:

diff --git a/config/derecho.yaml b/config/derecho.yaml
index 34b342e..ac20206 100644
--- a/config/derecho.yaml
+++ b/config/derecho.yaml
@@ -1,11 +1,11 @@
 machine:
   name: derecho
-  cores_per_node: 128
+  cores_per_node: 8
   head_node_name: derecho6
   scheduler:
     type: pbs
     account: p93300606
-    queue: main
+    queue: develop
     partition: None
 
 test:

Now that qstat -H is working on derecho again, testing ran more smoothly than last time. There were still some failures, as noted below. This time, there were no failures in performance tests or ESMF_CompTunnelUTest, though I expect that we could still see those failures sporadically in testing on these shared nodes. Perhaps more importantly, this time there were no mysterious issues of unreported test results like last time: it seems that the qstat fix resolved those issues.

gfortran-pio2.5.10-mpi-g

10 unit tests fail

CRASHED: mpi/g: src/Infrastructure/Geom/tests/ESMF_GeomUTest.F90
CRASHED: mpi/g: src/Infrastructure/XGrid/tests/ESMF_XGridMaskingUTest.F90

ESMF_GeomUTest has:

start failed on dec2433: No reply from shepherd after 108s

ESMF_XGridMaskingUTest has:

Couldn't attach to job 2400682.desched1: Unknown error
start failed on dec2433: Couldn't attach to job 2400682.desched1: Unknown error

gfortran-pio2.6.2-mpi-O

10 unit tests fail

CRASHED: mpi/O: src/Infrastructure/Mesh/tests/ESMCI_MeshMOABUTest.C

with:

start failed on dec2433: No reply from shepherd after 108s

intel-2023.2.1-oneAPI-mpi-O

1 system test fails

FAIL: mpi/O: src/system_tests/ESMF_FieldRegridLS/ESMF_FieldRegridLSSTest.F90

However, it looks like this has been failing for a while, so this isn't a new issue.

[billsacks]

At a glance at some of my recent charges, it does seem like the 8-core jobs to the develop queue are being charged much less than the jobs to the dedicated main queue, as expected. I'll do some spot-checks of apples-to-apples comparisons once I can get my recent job charges (it appears that the accounting is about a week behind).

I realized that we could potentially decrease the charges even further by using a single core for all of the mpiuni tests.

[billsacks]

In a hallway conversation with Ben Kirk today, he also supported the idea of moving at least some of our testing to casper, which I think has 32 and/or 64-core nodes, so at least would be somewhat cheaper than derecho. It has the same compilers as derecho but just has openmpi (not cray-mpi). He also offered to work with us on building out our GitHub Action-based testing if we'd like: he says that he already tests building ESMF (but not running the tests, I think) in a large variety of container-based environments: he uses ESMF as one of his test libraries to make sure he has an environment that will work for people.

[billsacks]

Here are some sample core-hour charges from recent jobs, comparing jobs run on in the main queue (with dedicated, 128-core nodes; note that these tested v8.9.0) vs. jobs run on the develop queue (using 8 cores of these shared nodes; note that these tested develop):

• gfortran-12.2.0_pio2.6.2-mpi-O:
  • main: 34.74 (build, 0.27 hours), 92.55 (test, 0.73 hours)
  • develop: 1.4 (build, 0.37 hours), 6.11 (test, 1.53 hours)
• gfortran-12.2.0_pio2.6.2-mpi-g
  • main: 33.42 (build, 0.26 hours), 109.26 (test, 0.86 hours)
  • develop: 1.36 (build, 0.35 hours), 6.65 (test, 1.66 hours)
• intel-2023.2.1-mpi-O
  • main: 31.04 (build, 0.24 hours), 93.65 (test, 0.73 hours)
  • develop: 1 (build, 0.26 hours), 6.23 (test, 1.56 hours)

Note that main is charged 128*time, develop only 4*time. Note that build and especially test runs slower on develop - by about a factor of 2 for test time. Should we have given twice as many cores on the develop queue??

Cost:

• build: about 25-30x more expensive on main
• test: about 16x more expensive on main, as expected
• total: about 17-18x more expensive on main