Skip to content

Create test config for cice5 integration #80

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
anton-seaice opened this issue Apr 1, 2025 · 59 comments
Open

Create test config for cice5 integration #80

anton-seaice opened this issue Apr 1, 2025 · 59 comments
Assignees

Comments

@anton-seaice
Copy link

Create a demo config for cice5 work, using ACCESS-NRI/ACCESS-ESM1.6#70 build and coupling updates

@anton-seaice
Copy link
Author

I put a start in here:

dev-preindustrial+concentrations...80-dev-preindustrial+concentrations+cice5

It needs some payu modifications to run ... I am working on those in payu-org/payu#572

@anton-seaice
Copy link
Author

I've done some runs with this configuration now. The configuration is based on the esm1.6 configuration from about March - similar to the PI-02 configuration. The labels are as follows:

  • cice5-db: cice5 with the parameters similar to @DaveBi
  • pi-02: PI-02 output on gadi
  • cice4 baseline to match the cice5 run from this repo
  • c5+bubbly: same as the other cice5 run but with 'bubbly' conductivity instead of 'MU71'

Image

Image

And compared to modern day Observations (purple):

Image

Southern hemisphere results don't look that good. I will try setting chio / sinw and cosw as suggested by @ofa001

I assume the initial conditions will not be impacting after a few years ... but should we be trying with some more realistic southern hemisphere initial conditions ?

@ofa001
Copy link

ofa001 commented Apr 16, 2025

Hi @anton-seaice, Yes the 1m initial conditions in the SH could be impacting these results still, especially the first few summers, but I am surprised the winters the ice extent is also greater than normal the difference between bubbly and MU71 is minor as I suggested.

The turning angle change may be small as well, it was tunings we had in cice4 but didnt include in cice5 as we had such large warm biases in SH and initially cold biases in our Arctic spin ups and didnt see how they would help.

Are these runs easy to spot in your tm70 space.

@anton-seaice
Copy link
Author

Here is the plot of SH volume:

Image

Summer sea-ice vol is similar between the runs, so i don't think its the initial conditions which are having a persistent impact? Unless we are saying the melt has cooled the ocean enough to do have a persistent impact ?)

The run with bubbly is here:

/g/data/tm70/as2285/payu/esm1.6-preind+c-cice5+bubbly

@anton-seaice
Copy link
Author

anton-seaice commented Apr 16, 2025

Some chats in the office agreed its possible that the initial conditions dumped so much freshwater into the ocean that its settled in quite a different state, so I will try a different initial condition.

OM2 initial conditions restarts don't run for some reason (errors in ITD redistribution, possibly related to change from BL99 to zero-layer thermodynamics). Ill try starting with 'none' initial condition, but we might need something better long term.

@ofa001
Copy link

ofa001 commented Apr 16, 2025 via email

@anton-seaice
Copy link
Author

I will check in the Southern Ocean how much fresher the ocean as got

Oh good - thanks. This will be interesting. We might not have enough ocean diagnostics turned on ...

its worked when we use it as an OM2 type initial condition and in runs that Noah and Dan have done with the standalone CICE6 set up, but there is no active ocean to get fresh in that situations.

sorry i wasn't very clear. I tried to start cice5 using a restart from the end of an access-om2 ryf run but leaving everything else unchanged. It loaded the restart but quickly crashed. I'm trying with no sea-ice to start but needs a few hours to run yet :)

@ofa001
Copy link

ofa001 commented Apr 16, 2025 via email

@ofa001
Copy link

ofa001 commented Apr 16, 2025

Hi @anton-seaice, The data was there in the ocean files, there is major melt of the 3m ice in the Southern Ocean in the first couple of months, then slower as the autumn/winter seasons occur so just at the ice edge as normal.

There is strong salinity signal in the surface 50m which gradually mixes down. If the initial condition was only 1m thick then we would only get 1/3 of the response and it would be less drastic, and we should get a better result I guess we used that back in cice4 (access-om, access 1-0/3) initialization period.

I will check the last year of that run to see how far the surface salinity signal has mixed, it will be through the main gyres, which is why the ice has travelled further north than usual.

@ofa001
Copy link

ofa001 commented Apr 16, 2025

Hi @anton-seaice just looked at year 10 of the run, the initial condition is still having an impact as even in the summer months the two gyres (Ross and Weddell for the non Oceanographers who are following) but particularly the Weddell are pulling the excess sea ice north its melting and cooling the waters and lowering the potential freezing point for the next winter, hence the signal is lasting and decaying as fast I would have expected, a 1m initial condidtion or none might work as an option. I will check how much ice was left in Dave's "none: initialized SH run by Dec31 and what the salinities were like in comparison to this run.

@ofa001
Copy link

ofa001 commented Apr 16, 2025

Hi @anton-seaice just looked at the end of year1 of Dave's "none' initialization run, there is still ice present in the SH. the salinity in the ocean freshens during the year but not as severely as in the 3m case so its worth trying as an option alongside 1m whilst keeping 3m for the Arctic as that is closer to the result we are aiming for there.
Also was just discussing with Dan his latest SH fast ice runs, he is initializing with the "no ice" condition which make sense for his case.

@anton-seaice
Copy link
Author

Volume looks much the same starting from no ice condition compared to 'default':

Image

Although areas look good (and possibly better) in the last 5 years of my not very long run:

Image

My feeling is to run that a bit longer as see what happens ?

I didn't have any success yet with starting with ~1m thick initial condition.

Config ran is here:

dev-preindustrial+concentrations...80-dev-preindustrial+concentrations+cice5

results are in the poorly named /g/data/tm70/as2285/payu/esm1.6-preind+c-cice5+om2_ic

@DaveBi
Copy link

DaveBi commented Apr 17, 2025

Hi @ofa001 @anton-seaice, obviously I made a realy poor mistake with the experiment setting -- forgot to switch "runtype" to "continue" after the first job ("initial") and thus the following years are all invalid. Thanks a lot for spotting this error. I've now tested the "continue" run but see the model crash because of " ice: Vertical thermo error" (again). This should not happen if the cice restart file is properly written and read in, and the restart fields are in right place the "continue" job. I wonder, Anton, if you have made any change to the relevant code for your sucessful tests...

@anton-seaice
Copy link
Author

anton-seaice commented Apr 17, 2025

Thanks @DaveBi

No particular code changes come to mind - it's running this commit - which is the same as your provided version with the fix for enum previously discussed. I've used the mom5/um7 versions from around march that were being used in esm1.6 but they should be fairly similar (to esm1.5).

The cice namelist I used is here - I think it scientifically similar to yours. I made the suggested changes from siobhan and set restart_ext to .false. but I don't think they are critical.

I did have to make sure the times in the namelist files were correct. Payu sets inidate in input_ice.nml, so I needed to turn off use_restart_time in cice_in.nml. I also needed to modify payu to set jobnum. I think the CSIRO run scripts will already be handling these parameters.

@anton-seaice
Copy link
Author

The rates in the cice5 output appear to work fine, e.g. circumpolar mean of change in volume due to thermodynamics (cm/day):

Image

which were broken in cice4

@anton-seaice
Copy link
Author

Native output to CMOR names appears to work ok too, but will need checking in detail :

Image

@anton-seaice
Copy link
Author

I am on leave next week, but there should be more results in /g/data/tm70/as2285/payu/esm1.6-preind+c-cice5+om2_ic after the weekend

Image

Image

We are definitely settling into a higher area and volume in the southern ocean, although possibly closer to modern day observations than the previous runs

Image

Image

@ofa001
Copy link

ofa001 commented Apr 22, 2025

Hi @anton-seaice We need to merge in the branch changes I had to the CMIP6 minor changes only but we can do that when you get back from a well earned break. I will check the ongoing runs it looks like starting from no ice is working Ok and we want the pre-industrial to be slightly above present day but not massively so.

@ofa001
Copy link

ofa001 commented Apr 24, 2025

Hi @anton-seaice I have had a look at your latest run when you started noice focussing on years 101 102 130 and 148 (end of run?) the ice in Arctic and Antarctic is acts fairly sensibly for a pre-industrial run a litle to the north of present day, maximum winter thickness in Southern Weddell sea of ~ 4m which circulates in the gyre and lasts the summer, Ross sea melts out. Arctic is alos a reasonable thickness for what we think of as pre-industrial ice thickness ( 1950's submarine ice conditions 3-4m central Arctic, more near Greenland and Canadian archipelago.

The freshening in the Southern Ocean is not as bad as in the earlier run, therier is still some melt atthe ice edge but some convection and deeper mixed layers in the centre of the Weddell sea, but not in the Ross. Convection shallower in the Greenland sea and N Atlantic, @DaveBi might want to run his diagnostics on the run if he has access. ( /g/data/tm70/as2285/payu/esm1.6-preind+c-cice5+om2_ic )

@anton-seaice
Copy link
Author

In a longer run, the Arctic ice are kept growing:

Image

SH climatology looks better than PI-02 in the longer run:

Image

but NH looks marginally worse than PI-02:

Image

It's not at all clear what is different between cice4 and 5 that would lead to the thicker ice, I had another look through the set-ups and couldn't see anything obvious to me.

@ofa001
Copy link

ofa001 commented Apr 28, 2025

Hi @anton-seaice I will check the namelists, and see if you aligned all the differences I saw earlier compared to cice4 set up, chatmore Wednesday, I looked at individual years of your runs back end of last week.

@anton-seaice
Copy link
Author

Hi @DaveBi

There seems to be significant drift in the global ocean volume in my run with cice5 (note the weird scale):

Image

I am not really sure how to diagnose that further !

@ofa001
Copy link

ofa001 commented May 1, 2025 via email

@DaveBi
Copy link

DaveBi commented May 1, 2025

@anton-seaice @ofa001 This drift does look a bit 'scaring' -- a increse of 0.05243%/50-years. Need to find out where the extra water comes from...
Anton,please can you run a paralell test setting add_lprec=0 in input_ice.nml?

@anton-seaice
Copy link
Author

there were lots of others some small at the decimal point level others constants larger but may have been overeiddiden by namelist choices. This ocean density of the ocean assumed by the ice model is also different in the two runs

In the source the density is set to this in both runs (is it logged in the output somewhere?):
rhow = 1026.0_dbl_kind ,&! density of seawater (kg/m^3)

I think MOM assumes 1035 however ?

CICE4 used:

cp_ocn    = 3989._dbl_kind   ,&! specific heat of ocn    (J/kg/K)
ice_ref_salinity = 4._dbl_kind ,&! (ppt)
ksno   = 0.30_dbl_kind  ,&! thermal conductivity of snow  (W/m/deg)

and CICE5 has been modified to this (for CM2 I guess)

cp_ocn    = 3992.10322329649_dbl_kind` 
ice_ref_salinity = 5._dbl_kind, &! (ppt)
ksno   = 0.2_dbl_kind  ,&! thermal conductivity of snow  (W/m/deg)

Shall we change these back to the CICE4 values?

There are also minor changes to the gravitational parameters. Let me know if you spot others i've missed in this post.

(Full diff ice_constants_F90_diff.txt)

@access-hive-bot
Copy link

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/coordinating-development-threads-for-next-phase-of-control-run/4523/1

@anton-seaice
Copy link
Author

After 59 years, I had this crash:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
um_hg3.exe         00000000013D3363  Unknown               Unknown  Unknown
libpthread-2.28.s  000014A00A76F990  Unknown               Unknown  Unknown
um_hg3.exe         000000000098C2C7  bi_linear_h_              472  bi_linear_h.f90
um_hg3.exe         0000000000D3629C  ritchie_                 2557  ritchie.f90
um_hg3.exe         0000000000A4C9ED  departure_point_          382  departure_point.f90
um_hg3.exe         00000000008AB30F  sl_thermo_                681  sl_thermo.f90
um_hg3.exe         00000000006ED216  ni_sl_thermo_             778  ni_sl_thermo.f90
um_hg3.exe         00000000004BDF45  Unknown               Unknown  Unknown
um_hg3.exe         00000000004368C0  Unknown               Unknown  Unknown
um_hg3.exe         0000000000415ADE  um_shell_                3930  um_shell.f90
um_hg3.exe         000000000040EC28  MAIN__                     40  flumeMain.f90
um_hg3.exe         000000000040EB62  Unknown               Unknown  Unknown
libc-2.28.so       000014A00A1BD7E5  __libc_start_main     Unknown  Unknown
um_hg3.exe         000000000040EA6E  Unknown               Unknown  Unknown

Is that the "normal" um crash ? Do I just re-run? (Or do i need to perturb it then re-run)

@DaveBi
Copy link

DaveBi commented May 20, 2025

@anton-seaice Yes, try perturbing the latest restart dump of UM and re-run the current job.

@anton-seaice
Copy link
Author

Thanks Dave - that worked.

Results are in:

/g/data/tm70/as2285/payu/esm1.6-preind-202504-cice5/archive

Ocean water volume looks ok in the new run (c5 is my run with cice5, c4 is Jhans April spin-up run, lines are annual mean/min/max):

Image

Areas and volume look ok. There's still a shift in sea-ice from northern to sourthern hemisphere (30-year climatologies from year 710 to 740):

Image

Image

Image

Image

@rml599gh
Copy link

@anton-seaice - are you OK if I grab a couple of these recent plots to show in a CMIP7 update at the ESM Working Group tomorrow. Results seem to be looking good.

@ofa001
Copy link

ofa001 commented May 21, 2025

Hi @anton-seaice Yes in these plots the cice5 looks a better match for the obs than the cice4, is there still something sifferent in the settings, Ice thickness distribution setting?? or did you change that. anyway it looks like cice 5 is better option which is why that option is the default for cice5.

@anton-seaice
Copy link
Author

Hi @ofa001 - I set kcatbound = 0 in this run. I think kcatbound = 1would be a better choice however?

@anton-seaice
Copy link
Author

Hi Rachel - sounds good, please do. Sorry they are a bit scrappy. Units are m2 / m3

@ofa001
Copy link

ofa001 commented May 21, 2025

Hi @anton-seaice Just checked they have the same, Ice thickness distributions now, (kcatbound), and all the other constants are now as aligned, I agree kcatbound makes more sense as you say above, kcatbound =0 is rather an old setting,

Puzzled why the two climatologies have diverged mostly in ice thickness and ice extent. Could just be variability and a longer run they will converge more. Both will have the same solar constant setting, as you started of Jhan's run set up, and I guess all his land changes.

@anton-seaice
Copy link
Author

Could just be variability and a longer run they will converge more.

Good suggestion, looks possible (annual max/mean/min) from the full lenght of the april-spinup:

Image

Image

Image

Image

@DaveBi
Copy link

DaveBi commented Jun 2, 2025

Evaluation of ocean and sea ice performance of this test run (@anton-seaice).

ESM1p5-6-eval.pptx

@DaveBi
Copy link

DaveBi commented Jun 2, 2025

Generally, CICE5 in ESM1.6 works well in terms of the resultant performance and ocean and sea ice. We see:

  1. "strong" overturning circulation off Antarctica (AABWF) and North Atlantic (NADWF) ;
  2. "Realistic" ACC transport;
  3. Relatively "reasonable" sea ice extent annual cycle (climatology); and even
  4. Improved ENSO period
  5. ...

@access-hive-bot
Copy link

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/csiro-access-nri-standup-minutes/3789/38

@DaveBi
Copy link

DaveBi commented Jun 3, 2025

@anton-seaice Can you please extend this test up to (at least) 200 years if ever possible? Thanks.

@anton-seaice
Copy link
Author

Yes will do. When do you want to look at the results? I'm hoping to get the changes merged into the access-esm1.6 github branch this week, and use that for extending the run. Is it ok if I start the run on friday ?

@ofa001
Copy link

ofa001 commented Jun 3, 2025

Hi @anton-seaice I am still intending looking to do another scan to see if I can find any other parameter differences left between Jhan's run and your run that can explain the difference in the result, that's present over 100year climatologies, it could just be the round off error stuff is cleaner in cice5, the ocean is less convective in the Weddell sea as well so the area is greater. Small things could be contributing, but it might be something that I missed.

@DaveBi
Copy link

DaveBi commented Jun 3, 2025

@anton-seaice yes, that's fine.
@ofa001 Have you been able to look at the ice diagnastic variables? Hope they are correctly produced as expected.
@rml599gh I've added Jhan's spinup run to the comparsion for the evulation. Attached slides show the ocean and sea ice performance. ESM1.6 with CICE5 yields results generally compariable to ESM1.6 with CICE4, and the ENSO "improvement" seems in deed results from CICE5 implementation.

ESM1p5-6-eval-20250603.pptx

@ofa001
Copy link

ofa001 commented Jun 3, 2025

Hi @DaveBi have looked at one or two of the key diagnostic variables we were worried about but not the whole list yet, Anton was going to do a run with some old switches on as a comparison I am not sure if that is the "demo" run.

Talk soon.

@DaveBi
Copy link

DaveBi commented Jun 4, 2025

@ofa001 @rml599gh @anton-seaice @har917 @access-hive-bot
The "issue" of ACC plots in my esm1.5/6 comparison is found:
The ACC transport is calculated using variables “TX_TRANS” and “TX_TRANS_GM” which are output in esm1.5 as monthly mean but in esm1.6 as yearly mean (COSIMA guys have made dramatic change to the diag_table and thus the ocean output format for esm1.6). So the ACC I got is monthly mean for esm1.5 and yearly mean for esm1.6. When I plot ACC, I do "l=@sbx:12" smoothing to get annual mean for both esm1.5 (which is roughly sensible) and esm1.6 (which is actually NOT needed anymore).
In the attached plot, ACC is smoothed with "l=@sbx:12" for esm1.5 but using the yearly mean value directly for esm1.6. They do look quite similarly "coarse" now, showing comparable interannual variablity.

Image

@anton-seaice
Copy link
Author

I will turn on the monthly tx_trans and tx_trans_gm diagnostics in my run. If you want these in the released configuration, then please make a new issue to capture that :)

@ofa001
Copy link

ofa001 commented Jun 4, 2025

Hi Anton, Just cross checked the last constants I was worried about, I think everything is now Ok, but I was a little confused which branch on the github I needed to look at the one you sent me a few weeks ago is no longer there who has been renamed. I cant keep up with those tags :(

See you have restarted the run at year zero.

@anton-seaice
Copy link
Author

Hi Siohbhan

Sorry - yes many complimentary things going on.

When I updated to by in sync with the "April" spin up, I create a new branch. See dev-preindustrial+concentrations...80-202504-dev-preindustrial+concentrations+cice5

I made a mistake yesterday with Payu which meant I ended up back a output000. The previous output (years 685-785) is here:

/g/data/tm70/as2285/esm1.6+cice5+apr

the new run (/g/data/tm70/as2285/payu/esm1.6-preind-202504-cice5/archive) starts from the 100th year (e.g. year 786) of that run

@ofa001
Copy link

ofa001 commented Jun 5, 2025

Thats OK @anton-seaice as long as the data was backed up, just spotted these changes here where you over wrote in the namelist the settings in the ice_constants which were still wrong which I spottet last night they did all flow through to the run didntt hey just loking how you have worded things in the code.

from your github comments
{expose ice_ref_salinity to the namelist (ACCESS-NRI/cice5#36)
This makes the assumed salinity of sea ice configurable, so we can set it the same as the MOM value

(configured in MOM here)

We use 4ppt in MOM for historical reasons, and did that in CICE4 too.}

From the CMIP5 cice4 history access-om had settings of ice-salinity of and access-cm of 5 due to their being a majority of 4-1 in the CSIRO team thinking the ice was more saline than MOM had chosen to use, value of 4 better suits old multiyear ice, which there isn't much of any more.

@anton-seaice
Copy link
Author

anton-seaice commented Jun 6, 2025

The run made it to year 809, and crashed with excessive salinity:

 The minimum T is    -2.904922427991E+00 deg C at (i,j,k) = (  29, 300,  4),  (lon,lat,dpt) = ( -279.4477,   74.9713,   35.0000 m)
 The grid dimensions are (dxt, dyt, dzt) = (    3.280811007397E+04,    3.182678360632E+04,    7.601028821708E+00 m)
 The grid dimensions (dst,rho_dzt) =     1.000000000000E+01,     7.866528197184E+03
 And the number of cells in the column are kmt =      5

 ...

 The maximum S is     5.523114930040E+01 psu at (i,j,k) = (  29, 300,  3),  (lon,lat,dpt) = ( -279.4477,   74.9713,   25.0000 m)
 The grid dimensions are (dxt, dyt, dzt) = (    3.280811007397E+04,    3.182678360632E+04,    7.601028821708E+00 m)
 The grid dimensions (dst,rho_dzt) =     1.000000000000E+01,     7.866528197184E+03
 And the number of cells in the column are kmt =      5
Variance temp =  2.83160268169910978E+01
|dT/dt|  temp =  1.20138814088956680E+00
Total    temp =  2.07899937338269152E+25

not really sure to make of that. That location is in the Arctic, and is under sea-ice, but there is nothing obviously strange with sea ice thickness or concentration. I guess I need to re-run with more frequent diagnostics ?

(crash is currently in /g/data/tm70/as2285/payu/esm1.6-preind-202504-cice5/work)

@ofa001
Copy link

ofa001 commented Jun 6, 2025

Hi @anton-seaice Anton, wasn't @ezhilsabareesh8 also finding strange high salinities in the Arctic, in some of the early om3 tests, I cant remember the reason it might have been shallow topography in his case, I don't ever
remember seeing it in ACCESS-ESM1.5 or its familay of models (ACCESS1-3 type) but I will have a look.

Went through and plotted after Dave's comment (above) about model output and the conversation on CMIP7 data request your run on cmor_data names still want to some cross checks with old ones we can do them next week through the namelist. We can talk more next week, but I will folow up this salinity issue and see if I can find @ezhilsabareesh8 example.

@ofa001
Copy link

ofa001 commented Jun 6, 2025

Hi @anton-seaice, and @DaveBi there was no partial ocean data in the payu work area saved for the year of the crash but the year before showed the region of ocean along that section of continental shelf was already at 44-45psu so its looks like its quickly gone off course depth in the region is 50m which I think is deeper, than was in the problem areas the @ezhilsabareesh8 had in his runs ( I will cross check, I think he had issues in the White Sea which is not far from this sector) we may have set a minimum depth for the Arctic shelves back when we set up the ACCESS1-0/3 topography. The salt flux from ice formation is high in these regions but not multiply times greater than in other Arctic regions as the new ice occurs in the Autumn.

@ofa001
Copy link

ofa001 commented Jun 6, 2025

Hi @anton-seaice I have noticed that this increase in high salinity on this Siberian shelf only occurs in your new run, there are some years that have been re-run after the 100year period, 787 onwards in the saved run, they don't have the problem this region of the ice is still fresh. Not sure what this means are some of the input files from the land side no longer inputting freshwater to the Arctic, which would be crucial in your restart as it was a fresh region of the Arctic so presumably from river runoff? (The area the @ezhilsabareesh8 had an issue in his test run was in the White sea so next sector along and was much shallower (11m) so slightly different issue and we had salinity relaxation as well, so probably not relevant to the current problem)

@ofa001
Copy link

ofa001 commented Jun 7, 2025

Hi @anton-seaice @MartinDix I was wondering last night if you altered anything to do with the salt_flux from the rivers after this comment from a couple of weeks back. Vertical Fresh water fluxes in MOM are handled as virtual salt fluxes so we may have still needed this flux if you switched it off? I havent checked how the Amazon region is fairing in your latest run?

#115 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

No branches or pull requests

5 participants