Skip to content

Support for high-resolution WACCM-X #1317

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: cam_development
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions bld/build-namelist
Original file line number Diff line number Diff line change
Expand Up @@ -3195,7 +3195,11 @@ if ($waccmx) {
add_default($nl,'ionos_xport_nsplit');
add_default($nl,'steady_state_ion_elec_temp', 'val'=>'.false.');
add_default($nl,'oplus_ring_polar_filter');
add_default($nl,'oplus_shapiro_const');
add_default($nl,'rxn_rate_sums');
if ($cfg->get('hgrid') =~ /ne120/) {
add_default($nl,'se_min_temperature', 'val'=>'105.D0');
}
}

# Chemistry options
Expand Down
4 changes: 3 additions & 1 deletion bld/configure
Original file line number Diff line number Diff line change
Expand Up @@ -1227,7 +1227,9 @@ if ($phys_pkg eq 'cam7') {
}
}
elsif ($waccmx) {
if ($phys_pkg eq 'cam6') {
if ($hgrid =~ /ne120/) {
$nlev = 273;
} elsif ($phys_pkg eq 'cam6') {
$nlev = 130;
} else {
$nlev = 126;
Expand Down
15 changes: 15 additions & 0 deletions bld/namelist_files/namelist_defaults_cam.xml
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,7 @@
<ncdata hgrid="ne120np4" nlev="30" aquaplanet="1" >atm/cam/inic/se/ape_cam5_ne120np4_L30_c170419.nc</ncdata>
<ncdata hgrid="ne120np4" nlev="32" >atm/cam/inic/se/F2000climo_ne120pg3_mt13_01-01-00000_c200420.nc</ncdata>
<ncdata hgrid="ne120np4" nlev="32" aquaplanet="1" >atm/cam/inic/se/ape_cam6_ne120np4_L32_c170908.nc</ncdata>
<ncdata hgrid="ne120np4" nlev="273" waccmx="1" >atm/waccm/ic/fx2000_ne120pg3L273.001.cam.i.0002-01-01-00000.noBRCL_c250522.nc</ncdata>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a reminder to deal with this file after it is discussed in the CSEG meeting


<ncdata hgrid="ne240np4" nlev="26" >atm/cam/inic/homme/cami_1850-01-01_ne240np4_L26_c110314.nc</ncdata>
<ncdata hgrid="ne240np4" nlev="26" ic_ymd="901" >atm/cam/inic/homme/cami_0000-09-01_ne240np4_L26_c061106.nc</ncdata>
Expand Down Expand Up @@ -880,19 +881,24 @@
<ionos_xport_nsplit hgrid="0.9x1.25" waccmx="1">30</ionos_xport_nsplit>
<ionos_xport_nsplit hgrid="ne30np4" waccmx="1">30</ionos_xport_nsplit>
<ionos_xport_nsplit hgrid="0.47x0.63" waccmx="1">90</ionos_xport_nsplit>
<ionos_xport_nsplit hgrid="ne120np4" waccmx="1">180</ionos_xport_nsplit>
<oplus_ring_polar_filter>.true.</oplus_ring_polar_filter>
<oplus_shapiro_const >3.d-2</oplus_shapiro_const>
<oplus_shapiro_const hgrid="ne120np4">6.d-3</oplus_shapiro_const>

<!-- Oplus transport grid configuration -->
<oplus_grid> 144,96 </oplus_grid>
<oplus_grid hgrid="ne16np4"> 144,96 </oplus_grid>
<oplus_grid hgrid="ne30np4"> 288,192 </oplus_grid>
<oplus_grid hgrid="0.9x1.25"> 288,192 </oplus_grid>
<oplus_grid hgrid="0.47x0.63"> 576,384 </oplus_grid>
<oplus_grid hgrid="ne120np4"> 576,384 </oplus_grid>
<!-- Magnetic grid resolution -->
<edyn_grid>80x97</edyn_grid>
<edyn_grid hgrid="0.9x1.25">160x193</edyn_grid>
<edyn_grid hgrid="ne30np4" >160x193</edyn_grid>
<edyn_grid hgrid="0.47x0.63">320x385</edyn_grid>
<edyn_grid hgrid="ne120np4" >320x385</edyn_grid>

<!-- ESMF mesh files for physics decomposition -->
<cam_physics_mesh hgrid="0.47x0.63">atm/cam/coords/fv0.47x0.63_esmf_c210305.nc</cam_physics_mesh>
Expand All @@ -905,6 +911,7 @@
<cam_physics_mesh hgrid="ne16np4" npg="3">share/meshes/ne16pg3_ESMFmesh_cdf5_c20211018.nc</cam_physics_mesh>
<cam_physics_mesh hgrid="ne30np4">atm/cam/coords/ne30np4_esmf_c210305.nc</cam_physics_mesh>
<cam_physics_mesh hgrid="ne30np4" npg="3">atm/cam/coords/ne30pg3_esmf_20200428.nc</cam_physics_mesh>
<cam_physics_mesh hgrid="ne120np4" npg="3">share/meshes/ne120pg3_ESMFmesh_cdf5_c20211018.nc</cam_physics_mesh>

<!-- For scaling lightning sources of NOx -->
<lght_no_prd_factor >1.00D0</lght_no_prd_factor>
Expand Down Expand Up @@ -1978,6 +1985,7 @@
<drydep_srf_file hgrid="ne60np4" >atm/cam/chem/trop_mam/atmsrf_ne60np4_110920.nc</drydep_srf_file>
<drydep_srf_file hgrid="ne120np4" >atm/cam/chem/trop_mam/atmsrf_ne120np4_110920.nc</drydep_srf_file>
<drydep_srf_file hgrid="ne120np4" npg="2">atm/cam/chem/trop_mam/atmsrf_ne120np4.pg2_200109.nc</drydep_srf_file>
<drydep_srf_file hgrid="ne120np4" npg="3">atm/cam/chem/trop_mam/atmsrf_ne120np4.pg3_c210324.nc</drydep_srf_file>
<drydep_srf_file hgrid="ne240np4" >atm/cam/chem/trop_mam/atmsrf_ne240np4_110920.nc</drydep_srf_file>

<drydep_srf_file hgrid="ne0np4CONUS.ne30x8" >atm/cam/chem/trop_mam/atmsrf_ne0np4conus30x8_161116.nc</drydep_srf_file>
Expand Down Expand Up @@ -3178,6 +3186,7 @@
<se_hypervis_subcycle hgrid="ne16np4" model_top="xt" > 18</se_hypervis_subcycle>
<se_hypervis_subcycle hgrid="ne30np4" model_top="ht" > 18</se_hypervis_subcycle>
<se_hypervis_subcycle hgrid="ne30np4" model_top="xt" > 18</se_hypervis_subcycle>
<se_hypervis_subcycle waccmx="1" hgrid="ne120np4" nlev="273" > 30</se_hypervis_subcycle>

<se_hypervis_subcycle_sponge > 1 </se_hypervis_subcycle_sponge>
<se_hypervis_subcycle_sponge hgrid="ne30np4" model_top="mt" > 3 </se_hypervis_subcycle_sponge>
Expand All @@ -3191,6 +3200,7 @@
<se_hypervis_subcycle_sponge hgrid="ne16np4" model_top="ht" > 2 </se_hypervis_subcycle_sponge>
<se_hypervis_subcycle_sponge hgrid="ne30np4" model_top="ht" > 4 </se_hypervis_subcycle_sponge>
<se_hypervis_subcycle_sponge hgrid="ne30np4" model_top="xt" > 40 </se_hypervis_subcycle_sponge>
<se_hypervis_subcycle_sponge waccmx="1" hgrid="ne120np4" nlev="273" > 180</se_hypervis_subcycle_sponge>

<se_hypervis_subcycle_q>1 </se_hypervis_subcycle_q>
<se_hypervis_subcycle_q hgrid="ne16np4">2 </se_hypervis_subcycle_q>
Expand Down Expand Up @@ -3225,13 +3235,16 @@
<se_molecular_diff waccmx="1" > 1.0 </se_molecular_diff>

<se_sponge_del4_nu_fac > -1 </se_sponge_del4_nu_fac>
<se_sponge_del4_nu_fac hgrid="ne120np4" nlev="273" waccmx="1"> 3.0 </se_sponge_del4_nu_fac>
<se_sponge_del4_nu_div_fac> -1 </se_sponge_del4_nu_div_fac>
<se_sponge_del4_nu_div_fac hgrid="ne16np4" waccm_phys="1" waccmx="0"> 7.5 </se_sponge_del4_nu_div_fac>
<se_sponge_del4_nu_div_fac model_top="ht" > 7.5 </se_sponge_del4_nu_div_fac>
<se_sponge_del4_nu_div_fac hgrid="ne120np4" nlev="273" waccmx="1" > 5.0 </se_sponge_del4_nu_div_fac>
<se_sponge_del4_lev > -1 </se_sponge_del4_lev>
<se_sponge_del4_lev waccm_phys="1"> 30 </se_sponge_del4_lev>
<se_sponge_del4_lev model_top="ht"> 30 </se_sponge_del4_lev>
<se_sponge_del4_lev model_top="xt"> 30 </se_sponge_del4_lev>
<se_sponge_del4_lev waccmx="1" hgrid="ne120np4" nlev="273"> 100 </se_sponge_del4_lev>

<se_qsplit > 1 </se_qsplit>
<se_qsplit hgrid="ne30np4" waccm_phys="1" waccmx="1" model_top="none"> 2 </se_qsplit>
Expand All @@ -3258,13 +3271,15 @@
<se_nsplit hgrid="ne0np4CONUS.ne30x8" waccm_phys="1" > 10 </se_nsplit>

<se_nsplit hgrid="ne0np4TESTONLY.ne5x4"> 7 </se_nsplit>
<se_nsplit waccmx="1" hgrid="ne120np4" nlev="273" > 10 </se_nsplit>

<se_rsplit > 3 </se_rsplit>
<se_rsplit waccm_phys="1" waccmx="0" > 2 </se_rsplit>
<se_rsplit model_top="ht" > 2 </se_rsplit>
<se_rsplit waccmx="1" > 4 </se_rsplit>
<se_rsplit hgrid="ne16np4" waccmx="1" model_top="none"> 2 </se_rsplit>
<se_rsplit hgrid="ne0np4CONUS.ne30x8" waccm_phys="1"> 3 </se_rsplit>
<se_rsplit waccmx="1" hgrid="ne120np4" nlev="273" > 5 </se_rsplit>

<se_fvm_supercycling>-1</se_fvm_supercycling>
<se_fvm_supercycling_jet>-1</se_fvm_supercycling_jet>
Expand Down
11 changes: 10 additions & 1 deletion bld/namelist_files/namelist_definition.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3226,7 +3226,7 @@ Default: .false.

<entry id="zmconv_parcel_hscale" type="real" category="conv"
group="zmconv_nl" valid_values="" >
The fraction of the boundary layer (PBL) depth, over which to mix the initial ZM convective parcel properties (fraction).
The fraction of the boundary layer (PBL) depth, over which to mix the initial ZM convective parcel properties (fraction).
Default: 0.5
</entry>

Expand Down Expand Up @@ -8191,6 +8191,15 @@ multiplied by a factor of se_molecular_diff.
Default: 0.
</entry>

<entry id="se_min_temperature" type="real" category="se"
group="dyn_se_inparm" valid_values="" >
Used by SE dycore to apply floor to temperature.....

NEEDS TO BE FILLED IN
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a reminder this does need to be filled in

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


Default: 0.
</entry>

<entry id="se_hypervis_subcycle_q" type="integer" category="se"
group="dyn_se_inparm" valid_values="" >
Number of hyperviscosity subcycles done in tracer advection code.
Expand Down
20 changes: 20 additions & 0 deletions cime_config/config_pes.xml
Original file line number Diff line number Diff line change
Expand Up @@ -542,6 +542,26 @@
</mach>
</grid>
<grid name="a%ne120">
<mach name="derecho">
<pes pesize="any" compset="_CAM\d0%WX">
<ntasks>
<ntasks_atm>-100</ntasks_atm>
<ntasks_lnd>-100</ntasks_lnd>
<ntasks_rof>-100</ntasks_rof>
<ntasks_ice>-100</ntasks_ice>
<ntasks_ocn>-100</ntasks_ocn>
<ntasks_cpl>-100</ntasks_cpl>
</ntasks>
<nthrds>
<nthrds_atm>1</nthrds_atm>
<nthrds_lnd>1</nthrds_lnd>
<nthrds_rof>1</nthrds_rof>
<nthrds_ice>1</nthrds_ice>
<nthrds_ocn>1</nthrds_ocn>
<nthrds_cpl>1</nthrds_cpl>
</nthrds>
</pes>
</mach>
<mach name="cheyenne">
<pes pesize="any" compset="_CAM6|_CAM7">
<comment>none</comment>
Expand Down
9 changes: 9 additions & 0 deletions cime_config/testdefs/testlist_cam.xml
Original file line number Diff line number Diff line change
Expand Up @@ -2332,6 +2332,15 @@
<option name="wallclock">00:20:00</option>
</options>
</test>
<test compset="FXHIST" grid="ne120pg3_ne120pg3_mg17" name="SMS_Lh1" testmods="cam/outfrq1h_wcmxhires">
<machines>
<machine name="derecho" compiler="intel" category="waccmx"/>
</machines>
<options>
<option name="wallclock">01:00:00</option>
<option name="comment">High-resolution WACCMX</option>
</options>
</test>
<test compset="FWmaHIST" grid="f09_f09_mg17" name="SMS_D_Ln9" testmods="cam/outfrq9s">
<machines>
<machine name="derecho" compiler="intel" category="waccm"/>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
./xmlchange ROF_NCPL=\$ATM_NCPL
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
mfilt=1,1,1,1,1,1
ndens=1,1,1,1,1,1
nhtfrq=-1,-1,-1,-1,-1,-1
write_nstep0 = .true.

avgflag_pertape= 'I', 'I', 'A', 'I', 'I'

empty_htapes=.true.
fincl1= 'Z3GM', 'T', 'U', 'V', 'OMEGA', 'ElecColDens', 'PS', 'O', 'O2', 'H',
'QRS_TOT', 'QRL_TOT', 'KVM', 'EKGW', 'TTGW', 'UTGW_TOTAL', 'VTGW_TOTAL'
fincl2= 'Z3GM', 'T', 'U', 'V', 'OMEGA', 'UI', 'VI', 'WI', 'EDens', 'PS', 'O', 'O2', 'H',
'OH', 'NO', 'O3', 'SIGMAHAL', 'SIGMAPED'
fincl3 ='T_24_COS', 'T_24_SIN', 'T_12_COS', 'T_12_SIN',
'U_24_COS', 'U_24_SIN', 'U_12_COS', 'U_12_SIN',
'V_24_COS', 'V_24_SIN', 'V_12_COS', 'V_12_SIN',
'QRS_TOT_12_SIN', 'QRS_TOT_12_COS'
fincl4 = 'ED1', 'ED2', 'PHIM2D', 'PHIHM', 'HALL_CONDUCTANCE', 'PED_CONDUCTANCE'
fincl5 = 'ElecColDens', 'PSL'
fincl6 = ' '
fincl7 = ' '
fincl8 = ' '
fincl9 = ' '
fincl10= ' '

ionos_npes = 3200
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
!----------------------------------------------------------------------------------
! Users should add all user specific namelist changes below in the form of
! namelist_var = new_namelist_value
!
! Include namelist variables for drv_flds_in ONLY if -megan and/or -drydep options
! are set in the CLM_NAMELIST_OPTS env variable.
!
! EXCEPTIONS:
! Set use_cndv by the compset you use and the CLM_BLDNML_OPTS -dynamic_vegetation setting
! Set use_vichydro by the compset you use and the CLM_BLDNML_OPTS -vichydro setting
! Set use_cn by the compset you use and CLM_BLDNML_OPTS -bgc setting
! Set use_crop by the compset you use and CLM_BLDNML_OPTS -crop setting
! Set spinup_state by the CLM_BLDNML_OPTS -bgc_spinup setting
! Set irrigate by the CLM_BLDNML_OPTS -irrig setting
! Set dtime with L_NCPL option
! Set fatmlndfrc with LND_DOMAIN_PATH/LND_DOMAIN_FILE options
! Set finidat with RUN_REFCASE/RUN_REFDATE/RUN_REFTOD options for hybrid or branch cases
! (includes $inst_string for multi-ensemble cases)
! Set glc_grid with CISM_GRID option
! Set glc_smb with GLC_SMB option
! Set maxpatch_glcmec with GLC_NEC option
! Set glc_do_dynglacier with GLC_TWO_WAY_COUPLING env variable
!----------------------------------------------------------------------------------
hist_nhtfrq = -1
hist_mfilt = 1
hist_ndens = 1

flanduse_timeseries = ' '
6 changes: 4 additions & 2 deletions src/dynamics/se/dycore/control_mod.F90
Original file line number Diff line number Diff line change
Expand Up @@ -121,11 +121,13 @@ module control_mod

!
! molecular diffusion
!
!
real(r8), public :: molecular_diff = -1.0_r8

integer, public :: vert_remap_uvTq_alg, vert_remap_tracer_alg


integer, public :: pgf_formulation = -1 !PGF formulation - see prim_advance_mod.F90

real(r8), public :: min_temperature = 0._r8

end module control_mod
14 changes: 13 additions & 1 deletion src/dynamics/se/dycore/prim_advance_mod.F90
Original file line number Diff line number Diff line change
Expand Up @@ -444,7 +444,7 @@ subroutine advance_hypervis_dp(edge3,elem,fvm,hybrid,deriv,nt,qn0,nets,nete,dt2,
use dimensions_mod, only: nu_scale_top,nu_lev,kmvis_ref,kmcnd_ref,rho_ref,km_sponge_factor
use dimensions_mod, only: nu_t_lev
use control_mod, only: nu, nu_t, hypervis_subcycle,hypervis_subcycle_sponge, nu_p, nu_top
use control_mod, only: molecular_diff,sponge_del4_lev
use control_mod, only: molecular_diff,sponge_del4_lev, min_temperature
use hybrid_mod, only: hybrid_t!, get_loop_ranges
use element_mod, only: element_t
use derivative_mod, only: derivative_t, laplace_sphere_wk, vlaplace_sphere_wk, vlaplace_sphere_wk_mol
Expand Down Expand Up @@ -688,6 +688,18 @@ subroutine advance_hypervis_dp(edge3,elem,fvm,hybrid,deriv,nt,qn0,nets,nete,dt2,
enddo
enddo
enddo
if (min_temperature>0._r8) then
Comment on lines 690 to +691
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was checking that being outside the loop over ic did not impact this new code. It isn't readily apparent to me what the loop over ic is actually doing, since the variable ic is not used in this subroutine. I assume that there must be something which is being advanced in biharmonic_wk_dp3d.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My change in within the substepping loop (within the ic loop). I assume the temperature floor should be applied each substep.

@PeterHjortLauritzen had some comments in the issue on where the temperature floor should be applied. I would like his input on this.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would follow DOE's implementation where the limiter is called in the last stage of the RK time-stepping:

https://github.com/E3SM-Project/E3SM/blob/28628343e7b5b25b9958e1ab601ef2b4e21a29ce/components/homme/src/theta-l/share/prim_advance_mod.F90#L326

That said, I don't know if it matters ... what we do know is that this seems to be an instability in this version of spetral-elements so putting it in the time-stepping of the inviscid equations of motion seems like a good place ...

More details from DOE here:

E3SM-Project/E3SM#5089

! apply floor to temperature
do ie=nets,nete
do k=sponge_del4_lev+2,nlev
do j=1,np
do i=1,np
elem(ie)%state%T(i,j,k,nt) = max(elem(ie)%state%T(i,j,k,nt),min_temperature)
end do
end do
end do
end do
end if
call tot_energy_dyn(elem,fvm,nets,nete,nt,qn0,'dAH')
end do

Expand Down
15 changes: 14 additions & 1 deletion src/dynamics/se/dyn_comp.F90
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@ subroutine dyn_readnl(NLFileName)
use control_mod, only: max_hypervis_courant, statediag_numtrac,refined_mesh
use control_mod, only: molecular_diff, pgf_formulation, dribble_in_rsplit_loop
use control_mod, only: sponge_del4_nu_div_fac, sponge_del4_nu_fac, sponge_del4_lev
use control_mod, only: min_temperature
use dimensions_mod, only: ne, npart
use dimensions_mod, only: large_Courant_incr
use dimensions_mod, only: fvm_supercycling, fvm_supercycling_jet
Expand Down Expand Up @@ -173,6 +174,8 @@ subroutine dyn_readnl(NLFileName)
real(r8) :: se_molecular_diff
integer :: se_pgf_formulation
integer :: se_dribble_in_rsplit_loop
real(r8) :: se_min_temperature = 0.0_r8

namelist /dyn_se_inparm/ &
se_fine_ne, & ! For refined meshes
se_ftype, & ! forcing type
Expand Down Expand Up @@ -218,7 +221,8 @@ subroutine dyn_readnl(NLFileName)
se_kmax_jet, &
se_molecular_diff, &
se_pgf_formulation, &
se_dribble_in_rsplit_loop
se_dribble_in_rsplit_loop, &
se_min_temperature
!--------------------------------------------------------------------------

! defaults for variables not set by build-namelist
Expand Down Expand Up @@ -293,6 +297,8 @@ subroutine dyn_readnl(NLFileName)
call MPI_bcast(se_molecular_diff, 1, mpi_real8, masterprocid, mpicom, ierr)
call MPI_bcast(se_pgf_formulation, 1, mpi_integer, masterprocid, mpicom, ierr)
call MPI_bcast(se_dribble_in_rsplit_loop, 1, mpi_integer, masterprocid, mpicom, ierr)
call MPI_bcast(se_min_temperature, 1, mpi_real8, masterprocid, mpicom, ierr)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nusbaume - Probably next time we are in the SE dycore code, we should add checks for the MPI calls


if (se_npes <= 0) then
call endrun('dyn_readnl: ERROR: se_npes must be > 0')
end if
Expand Down Expand Up @@ -361,6 +367,8 @@ subroutine dyn_readnl(NLFileName)
molecular_diff = se_molecular_diff
pgf_formulation = se_pgf_formulation
dribble_in_rsplit_loop = se_dribble_in_rsplit_loop
min_temperature = se_min_temperature

if (fv_nphys > 0) then
! Use finite volume physics grid and CSLAM for tracer advection
nphys_pts = fv_nphys*fv_nphys
Expand Down Expand Up @@ -496,6 +504,11 @@ subroutine dyn_readnl(NLFileName)
se_write_restart_unstruct

write(iulog, '(a,e9.2)') 'dyn_readnl: se_molecular_diff = ', molecular_diff

if (min_temperature>0._r8) then
write(iulog, '(a,e9.2)') 'dyn_readnl: se_min_temperature = ', min_temperature
end if

end if

call native_mapping_readnl(NLFileName)
Expand Down