Labeling new fields in UPP

[Jeff-Duda]

I have added a few new fields to be output from MPAS to also be written to GRIB2 files from UPP. These new fields are

1. hourly-maximum composite reflectivity
2. hourly-maximum precipitation rate
3. hourly-maximum 1-minute average precipitation rate
4. hourly-maximum 5-minute average precipitation rate
5. hourly-maximum 10-minute average precipitation rate

Item #2 in the list above already exists in the UPP, so outputting that is not a problem. I also have modified the code to read the remaining arrays and dump them into GRIB2 output as well. Since these fields are all fully calculated in MPAS, they are just passed through UPP.

The problem arrives in labeling these arrays so that they are easily distinguished when browsing the contents of GRIB2 files. I'm mainly referring to using wgrib2 to query the contents of a file.

For example, I processed a subset of fields from an MPASSIT output file that contains the fields above as well as instantaneous composite reflectivity and precipitation rate. Here's what the wgrib2 -v output looks like:

1:0:d=2025071400:REFC Composite reflectivity [dB]:entire atmosphere (considered as a single layer):1 hour fcst:
2:644383:d=2025071400:REFC Composite reflectivity [dB]:entire atmosphere (considered as a single layer):1 hour fcst:
3:1172847:d=2025071400:MAXREF Hourly Maximum of Simulated Reflectivity [dB]:1000 m above ground:0-1 hour max fcst:
4:1496560:d=2025071400:REFC Composite reflectivity [dB]:entire atmosphere (considered as a single layer):1 hour fcst:
5:1496748:d=2025071400:PRATE Precipitation Rate [kg/m^2/s]:surface:0-1 hour max fcst:
6:1496960:d=2025071400:PRATE Precipitation Rate [kg/m^2/s]:surface:0-1 hour max fcst:
7:1607268:d=2025071400:PRATE Precipitation Rate [kg/m^2/s]:surface:0-1 hour max fcst:
8:1666332:d=2025071400:PRATE Precipitation Rate [kg/m^2/s]:surface:0-1 hour max fcst:

You'll notice that there are several arrays with identical descriptors. Therefore the only way to determine which REFC record is the time-maximum or instantaneous is to plot them, which is less than ideal. I want to know how I should define the various GRIB2 attributes so as to easily distinguish between these arrays. I have visited with @jaymes-kenyon at GSL who has suggested that perhaps new GRIB2 table variable names should be created for these arrays. For time-max composite reflectivity it seems natural to consider MXREFC (akin to MXUPHL for time-max updraft helicity), whereas for the sub-hourly averaged precipitation rates, something like PRATE1, PRATE5, etc. may work.

[BenjaminBlake-NOAA]

@Jeff-Duda I agree that new GRIB2 table variable names would be the best way to handle this issue. I'm the EMC point of contact for the GRIB2 tables and can work with you on getting them updated. I'm participating in the FFaIR experiment this week but will have more time to work on this next week.

An hourly maximum composite reflectivity field could certainly be added to Table 4.2-0-16. I think I have a slight preference for MAXREFC instead of MXREFC because we already have MAXREF defined for the hourly max simulated reflectivity.

For the hourly max precip rates, adding PRATE1, PRATE5 to Table 4.2-0-1 could work, but I'm also wondering if it would be better to define a new "MAXPRATE" variable. I will need to think about what is best.

[WenMeng-NOAA]

hourly-maximum composite reflectivity

Right now the UPP only output instant REFC, so a new entry can be add in post_avblflds.xml for hourly-maximum REFC, such as

    <param>
       <post_avblfldidx>XXXX</post_avblfldidx>
       <shortname>MAX_REFC_ON_ENTIRE_ATMOS</shortname>
      <pdstmpl>tmpl4_8</pdstmpl>
       <pname>REFC</pname>     
       <stats_proc>MAX</stats_proc>
       <fixed_sfc1_type>entire_atmos_single_lyr</fixed_sfc1_type>
       <scale>4.0</scale>
    </param>

[BenjaminBlake-NOAA]

@WenMeng-NOAA Thanks for the suggestion, that is true and we could simply add this new entry to post_avblflds.xml for hourly-max REFC, which would label it as:
REFC Composite reflectivity [dB]:entire atmosphere (considered as a single layer):0-1 hour max fcst:

That is what's done for hourly max PRATE now.

    <param>
       <post_avblfldidx>508</post_avblfldidx>
       <shortname>MAX_PRATE_ON_SURFACE</shortname>
       <longname>Maximum Precipitation Rate on surface</longname>
       <pdstmpl>tmpl4_8</pdstmpl>
       <pname>PRATE</pname>
       <stats_proc>MAX</stats_proc>
       <fixed_sfc1_type>surface</fixed_sfc1_type>
       <scale>6.0</scale>
    </param>

I think I still have a slight preference to add a new GRIB2 variable MAXREFC though since we have a MAXREF variable defined.

[WenMeng-NOAA]

hourly-maximum 1-minute average precipitation rate
hourly-maximum 5-minute average precipitation rate
hourly-maximum 10-minute average precipitation rate

As @BenjaminBlake-NOAA mentioned, these three fields could be solved via adding new grib2 names in Table 4.2-0-1. Since they are maximum values, I would suggest to use max definition entry, such as

    <param>
       <post_avblfldidx>XXXX</post_avblfldidx>
       <shortname>1M_MAX_PRATE_ON_SURFACE</shortname>
       <pdstmpl>tmpl4_8</pdstmpl>
       <pname>NEW_NAME</pname>
       <stats_proc>MAX</stats_proc>
       <fixed_sfc1_type>surface</fixed_sfc1_type>
       <scale>6.0</scale>
    </param>

In table 4-2-0-1, you might clarify these new grib2 name as 1-minute average, 5-minutes, 10-minute average.

[WenMeng-NOAA]

@WenMeng-NOAA Thanks for the suggestion, that is true and we could simply add this new entry to post_avblflds.xml for hourly-max REFC, which would label it as: REFC Composite reflectivity [dB]:entire atmosphere (considered as a single layer):0-1 hour max fcst:

That is what's done for hourly max PRATE now.

    <param>
       <post_avblfldidx>508</post_avblfldidx>
       <shortname>MAX_PRATE_ON_SURFACE</shortname>
       <longname>Maximum Precipitation Rate on surface</longname>
       <pdstmpl>tmpl4_8</pdstmpl>
       <pname>PRATE</pname>
       <stats_proc>MAX</stats_proc>
       <fixed_sfc1_type>surface</fixed_sfc1_type>
       <scale>6.0</scale>
    </param>

I think I still have a slight preference to add a new GRIB2 variable MAXREFC though since we have a MAXREF variable defined.

@BenjaminBlake-NOAA Your preference is fine for me.

[BenjaminBlake-NOAA]

@WenMeng-NOAA Sounds good, thanks for the information on updating post_avblflds.xml.

@Jeff-Duda @jaymes-kenyon After I update the GRIB2 tables online, I can provide updated params_grib2_tbl_new and post_avblflds.xml files for you. Then you can update the UPP control files for MPAS to include the new fields and run a test to confirm they are working as intended. If the changes work well for you then you can go ahead and open a UPP PR.

[Jeff-Duda]

@WenMeng-NOAA and @BenjaminBlake-NOAA, thank you both for the quick responses!

I'm glad to hear that you're willing to modify the GRIB2 tables for this, as it makes my job much easier.

I'll note that I have already modified the code heavily for this work. In fact, I have updated the post_avblflds.xml file accordingly as well to add

For time-max composite reflectivity

<param>
   <post_avblfldidx>244</post_avblfldidx>
   <shortname>MAX_REFC_ON_ENTIRE_ATMOS</shortname>
   <longname>time-maximum composite reflectivity</longname>
   <pname>REFC</pname>
   <stats_proc>MAX</stats_proc>
   <fixed_sfc1_type>entire_atmos_single_lyr</fixed_sfc1_type>
   <scale>4.0</scale>
</param>

and, for the sub-hourly precip rates:

<param>
   <post_avblfldidx>518</post_avblfldidx>
   <shortname>MAX_PRATE_1MIN_ON_SURFACE</shortname>
   <longname>Maximum 1-minute average precipitation rate on surface</longname>
   <pdstmpl>tmpl4_8</pdstmpl>
   <pname>PRATE</pname>
   <stats_proc>MAX</stats_proc>
   <fixed_sfc1_type>surface</fixed_sfc1_type>
   <scale>6.0</scale>
</param>

<param>
   <post_avblfldidx>519</post_avblfldidx>
   <shortname>MAX_PRATE_5MIN_ON_SURFACE</shortname>
   <longname>Maximum 5-minute average precipitation rate on surface</longname>
   <pdstmpl>tmpl4_8</pdstmpl>
   <pname>PRATE</pname>
   <stats_proc>MAX</stats_proc>
   <fixed_sfc1_type>surface</fixed_sfc1_type>
   <scale>6.0</scale>
</param>

<param>
   <post_avblfldidx>520</post_avblfldidx>
   <shortname>MAX_PRATE_10MIN_ON_SURFACE</shortname>
   <longname>Maximum 10-minute average precipitation rate on surface</longname>
   <pdstmpl>tmpl4_8</pdstmpl>
   <pname>PRATE</pname>
   <stats_proc>MAX</stats_proc>
   <fixed_sfc1_type>surface</fixed_sfc1_type>
   <scale>6.0</scale>
</param>

Furthermore, given that the open spaces for the numerical field ids (post_avblfldidx) is filling up, I also added a comment to the top of the file to list the remaining available index values, i.e.:

<?xml version="1.0"?>
<postxml xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="EMC_POST_Avblflds_Schema.xsd">
<!--  Lin Gan:  20150324 validated -->

<!-- list of remaining unclaimed ID numbers:
 248 
 414 
 438 
 439 
 440 
 441 
 442 
 478 
 479 
 481 
 509 
 521 
 522 
 523 
 524 
 541 
 542 
 543 
 544 
 545 
 550 
 554 
 558 
 562 
 589 
 590 
 591 
 592 
 593 
 594 
 595 
 596 
 597 
 598 
 599 
 714 
...
....
....
--!>

One last question: there are files named
grib2_params_tbl_new
and
grib2_params_tbl_new.txt

They have different text formats, but do both need to be updated or just one? Does one overwrite the other when something runs?

[BenjaminBlake-NOAA]

@Jeff-Duda Sounds good! I'll provide you with updated versions of both the params_grib2_tbl_new and params_grib2_tbl_new.txt files. The params_grib2_tbl_new is the only one that is read in by UPP, and the .txt version is merely for documentation purposes, but both will need to be updated in UPP with these changes.

[BenjaminBlake-NOAA]

@Jeff-Duda @WenMeng-NOAA I've updated Table 4.2-0-16 to add MAXREFC (parameter 199) and Table 4.2-0-1 to add PRATE1MIN, PRATE5MIN, and PRATE10MIN (parameters 238, 239, 240). Let me know if you have any concerns with these parameter names.

If not, I have the updated params_grib2_tbl_new and params_grib2_tbl_new.txt files for you here:
https://ftp.emc.ncep.noaa.gov/mmb/bblake/files_for_jeff/

Please test these with your code changes. Since g2tmpl will also have to be updated, wgrib2 won't recognize the new parameter names yet. Thanks!

[WenMeng-NOAA]

@BenjaminBlake-NOAA The grib2 table updates look good to me.

[Jeff-Duda]

I made the changes indicated and re-ran, but I get these errors from the grib2_module.f file:

 WRFPOST:  PREPARE TO PROCESS NEXT GRID
 WRONG, could not find MAXREFC in WMO and NCEP table!
 WRONG, could not find MAXREFC in WMO and NCEP table!
  discipline    0  category    1  parameter    7 for var PRATE
 WRONG, could not find PRATE5MIN in WMO and NCEP table!
  discipline    0  category   16  parameter  196 for var REFC
  discipline    0  category   16  parameter  198 for var MAXREF
 WRONG, could not find PRATE10MIN in WMO and NCEP table!
 WRONG, could not find PRATE1MIN in WMO and NCEP table!
  discipline    0  category    1  parameter    7 for var PRATE

I checked that I have properly linked the updated tables to my run directory, so that doesn't seem like the problem. What else should I be looking to change?

[BenjaminBlake-NOAA]

@Jeff-Duda In your post_avblflds.xml file, you'll need to update your <pname> values to MAXREFC, PRATE1MIN, PRATE5MIN, and PRATE10MIN for the new grib2 variables. In the new entries you shared above they are using REFC and PRATE, which would explain why it couldn't find MAXREFC, PRATE1MIN, etc. After updating post_avblflds.xml with the new parameter names, you'll also need to re-generate the postxconfig text file.

I tried testing this myself and I was able to get MAXREFC in my output file, but not the hourly max PRATE fields. Maybe they are missing from my MPAS forecast output file since they are new fields? MAXREFC is labeled as the following, which is almost correct - I think it should be 1-2 hour max fcst and not 2 hour fcst:
var discipline=0 center=7 local_table=1 parmcat=16 parm=199:entire atmosphere (considered as a single layer):2 hour fcst:

I've included my updated UPP control files and my POSTTWO.tm00 file in this directory if they would be helpful:
https://ftp.emc.ncep.noaa.gov/mmb/bblake/files_for_jeff/

[Jeff-Duda]

@BenjaminBlake-NOAA Thanks. I'm pretty sure I have done all that, and I even tried rebuilding UPP, but the error remains. I have attached relevant files here to assist with troubleshooting.

I should also mention that the step to validate the post_avblflds.xml file fails. I'm talking about this command:

xmllint --noout --schema EMC_POST_Avblflds_Schema.xsd post_avblflds.xml

It brings up A LOT of error lines. This occurred with the file in my local repo and with the one in your FTP directory.

post_avblflds.xml.txt
postxconfig-NT-test.txt
UPP_f01.o25488856.txt