Derecho scaling issue: cesm1.3

[jedwards4b]

I am getting really poor scaling from an ne120 FC5 using the ihesp version of cam on derecho.
Scaling plot is here. I have traced the issue to phys_grid.F90:transpose_block_to_chunk

There are several options to control this routine including lbal_opt and a CPPDEF MODCM_DP_TRANSPOSE
Can someone suggest what options I should explore first?

[gold2718]

Does the ihesp version of CAM use the new infrastructure mods that CAM-SE uses?
The new version does not even have this routine.

[gold2718]

It came in with the cam6_3_007 tag.

[jedwards4b]

@gold2718 I'm not sure what you mean when you say that the new version does not have this routine.
Here is a pointer to the code in cam-development, it looks very similar to the code in the ihesp version.

[gold2718]

Yes but if you look at bld/configure, you will see that that code is not compiled for CAM-SE or CAM-MPAS.
Instead, the version in src/infrastructure shadows that code and scales much better in time and memory.

[jedwards4b]

okay - the ihesp code does not contain that. Thank you for clarifying.

[jedwards4b]

Changing the lbal_opt from 2 to 1 made a difference. I'm not sure that updating the ihesp code to use the latest cam is realistic - is it?

[gold2718]

I would have to review cam6_3_007 (it's been a while) but I think most of the impact was in the dycore.
However, looking at the ChangeLog, I see there were lots of files that used the old interfaces (the serial ones from physics/cam/phys_grid.F90) so those would have to be modified as well (see notes about use of deprecated interfaces).

[jedwards4b]

I don't suppose anyone has a program that isolates and tests the performance of lbal_opt in cam?

[gold2718]

No but I always found that phys_loadbalance = 0 was best for CAM-SE.