add state NaN checker

[juliasloan25]

Purpose

closes #969

Adds a function that quantitatively checks how many NaNs are present in the state, and displays this information to the user. This can be used to inspect sol.u[end] after a simulation has run, to see if any NaNs were produced at the end of the simulation.

If no NaNs are found, this information is logged in an info statement. If NaNs are found, this information is logged in a warn statement.

To-do

• add function to recursively check NaNs in state
• add tests
• add optional mask argument
• add monthly callback to longruns

[kmdeck]

do you want to also add this to our global runs in this PR? soil, soil canopy, soil canopy snow? perhaps we can add it as a callback to check every month e.g.

[juliasloan25]

Looking at the snowy land longrun here, we see that from months 1-5, only Y.soilco2.C contains NaNs, the number of which increases from 2516460 to 2800335 during this period. At month 6, NaNs also appear in the soil, canopy, and snow sections of the state, but much fewer than in the soilco2 model (60, 4, and 4 NaNs in each of the state variables of these models, respectively). Edit: note that soil has 15x the number of NaNs as canopy and snow because of the 15 subsurface levels.

I also interactively checked the state immediately after initialization, and there are no NaNs in any of the state, including the soilco2 model.

I'm not sure what to make of this but I'll continue looking into it. Investigating these NaNs is separate from the purpose of this PR, which is to provide the infrastructure to be able to see the NaNs, so this doesn't need to block merging this PR.

[kmdeck]

Thanks, Julia! this will be super helpful.

[kmdeck]

Looking at the snowy land longrun here, we see that from months 1-5, only Y.soilco2.C contains NaNs, the number of which increases from 2516460 to 2800335 during this period. At month 6, NaNs also appear in the soil, canopy, and snow sections of the state, but much fewer than in the soilco2 model (60, 4, and 4 NaNs in each of the state variables of these models, respectively).

I also interactively checked the state immediately after initialization, and there are no NaNs in any of the state, including the soilco2 model.

I'm not sure what to make of this but I'll continue looking into it. Investigating these NaNs is separate from the purpose of this PR, which is to provide the infrastructure to be able to see the NaNs, so this doesn't need to block merging this PR.

soil co2 has problems - we dont check it in any diagnostic or map right now, so we arent catching the NaNs. it's something @AlexisRenchon and I need to go back to.

could some of the NaNs be over the ocean?

agreed that that is a separate problem!

[juliasloan25]

Looking at the snowy land longrun here, we see that from months 1-5, only Y.soilco2.C contains NaNs, the number of which increases from 2516460 to 2800335 during this period. At month 6, NaNs also appear in the soil, canopy, and snow sections of the state, but much fewer than in the soilco2 model (60, 4, and 4 NaNs in each of the state variables of these models, respectively).
I also interactively checked the state immediately after initialization, and there are no NaNs in any of the state, including the soilco2 model.
I'm not sure what to make of this but I'll continue looking into it. Investigating these NaNs is separate from the purpose of this PR, which is to provide the infrastructure to be able to see the NaNs, so this doesn't need to block merging this PR.

soil co2 has problems - we dont check it in any diagnostic or map right now, so we arent catching the NaNs. it's something @AlexisRenchon and I need to go back to.

could some of the NaNs be over the ocean?

agreed that that is a separate problem!

It could be over the ocean, but from the info we get from this checker it's hard to tell.

Before these runs, I expected we would see NaNs over the ocean in most (all?) variables after 1 timestep. But I guess even over the ocean things can be stable for some time and only break later on. If this is the case, I'm not sure how helpful the single number of NaNs we provide will be, since it's impossible to distinguish if they're over ocean or not. We could solve this by including the mask in our check though.