NaNs detected in repro sum input #7271
-
|
Hello everyone, Any help is appreciated. Eamxx used to work fine with the parameters I gave it. Then I edited the code to add the kokkos profiling library, names to parallel_for that support it, and region names. This used to work still. Then while I was trying to profile with NVIDIA tools, something changed maybe in the underlying software versions in the system and I see the following. I can't make sense of it so any help would be great. Thank you in advance. the srun command is: |
Beta Was this translation helpful? Give feedback.
Answered by
mahf708
Apr 23, 2025
Replies: 9 comments 26 replies
-
|
@mihelog I edited your post slightly for better readability (code blocks) Is this error reproducible? If so: additionally, how are you triggering this command? Are you on a compute node with GPUs? If so, please make sure of the two following items: First: that you have only activate the corresponding run environment, i.e., Second: make the srun command GPU-aware with cc @ndkeen |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
[moving to discussion since our extensive testing didn't detect this issue, so it is likely a user-side config issue that we can continue discussing] |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
First off, thank you SO MUCH for your quick responses! I'm grateful for that. Likewise, thanks so much in advance for any further help. I'm invoking through a complicated script made for my local system that used to work. I don't know if it's my changes that broke it or not, or some system update. Here's what I did: 1). I added "-G 4" to the srun invocation. No change. Also, the sbatch script that ran does have " --gpus-per-node=4". This is what it has to be exact: 2). I tried invoking it through a "clean" interactive session but that threw another error, likely unrelated due to that environment, but FYI it's here " ERROR: (cime_cpl_init) :: namelist read returns an end of file or end of record condition". 3). I cleaned up my environment variables to only system defaults plus these two user-defined ones: 4). I confirmed that the second file you pointed to above runs so the environment variables specified in it are loaded. Likewise, all the modules in that script are loaded. The following are in addition, that are system defaults: |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Can you share the full script? I think the number 2 error may be pretty straightforward, you need to be in the run directory ($case_dir/run). |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, so you're not triggering srun directly. If you're doing it through sbatch, then you don't need the -G4 in the srun command (it will carry over). I think the most productive thing is to share your full script next. Otherwise, this could be a lot of different things :/ |
Beta Was this translation helpful? Give feedback.
-
|
Agreed. I'm invoking a custom script (not written by me unfortunately) that uses sbatch to invoke sbatch that includes the srun command. I'm puzzled here because it did used to work and broke at some point and remained broken even after reverting changes. What I did was add some kokkos profiling region names in the source code, linked against the kokkos library, and added nvidia profiling tools to srun. That worked well but even after removing them it didn't fix anything. So who knows. This is the srun command that ends up being executed through sbatch as a reminder: Anyway thanks in advance again. Here is the script. |
Beta Was this translation helpful? Give feedback.
9 replies
-
|
@ndkeen thank you so much for that script :) It's a lifesaver! Sure let me try that. That'll overwrite some of my source code changes but it's ok I can restore them. I can confirm that the error happens just fine without profiling. How I added profiling is by editing cime/CIME/case/case_run.py to do some string editing into the srun command to add the invocation to NVIDIA's tools. But after I remove those the problem persists. So I can't find correlation between the error and profiling. Edit: I saw you said with a new case name. I did give it a new case name and I'm not sure about the E3SM directory: |
Beta Was this translation helpful? Give feedback.
-
|
Let's try a new code root too: |
Beta Was this translation helpful? Give feedback.
-
|
Still doesn't like something. |
Beta Was this translation helpful? Give feedback.
-
|
oh dang ... I didn't realize you were trying to run the scream repo!!! That may explain it (that repo is no longer maintained) - local repo=scream
+ local repo=E3SMThe code and all dev is now in E3SM proper |
Beta Was this translation helpful? Give feedback.
-
|
If that doesn't work, please clone this repo (E3SM-Project/E3SM) then do |
Beta Was this translation helpful? Give feedback.
-
|
I see what you mean. But it's the new repo I was using. When my collaborator gave me the script, the fetch function was already inactive (do_fetch_code=false) and I pulled the new repo manually. So that should not be a problem. But it's an indication that maybe the script I'm using is old. Apologies if I'm missing something basic btw. When I fetch and update submodules, I get the error at the end of this post when running the script. I tried with several python versions (all 3.x though). Maybe my script is old too? When I restore my script edits in the "cime" directory, nothing changes which makes sense because I commented out all my changes. By doing a diff between the version of the "eamxx" directory I was using and this new one, there are many changes. So it's possible I was using an older version. Could it be an input file problem too? I'm including below the input .yaml file just in case YAML FILE THE ERROR |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Try this: output yaml
%YAML 1.1
---
averaging_type: average
fields:
physics_pg2:
field_names:
# 2D
- precip_liq_surf_mass_flux # rainfall rate
- precip_ice_surf_mass_flux # snow/ice rate
- RainWaterPath
- LiqWaterPath
- IceWaterPath
- VapWaterPath
- LW_flux_up_at_model_top # OLR
- SeaLevelPressure # SLP
- ps # surface pressure
- T_2m # 2m air temperature
- ZonalVapFlux
- MeridionalVapFlux
- wind_speed_10m # 10m wind speed
- U_at_10m_above_surface # zonal wind at 10m
- V_at_10m_above_surface # meridional wind at 10m
- landfrac # land sea mask
max_snapshots_per_file: 8 # one file per day
restart:
force_new_file: true
filename_prefix: scream_output.3hourlyAVG.h
iotype: pnetcdf
output_control:
frequency: 3 # output frequency
frequency_units: nhours # output frequency units (e.g. nmins, nhours)
flush_frequency: 1 # this is to ensure files that are written are flushed immediately -- otherwise might be lostSorry this was changed recently (mainly to unify naming scheme, etc.) |
Beta Was this translation helpful? Give feedback.
-
|
It doesn't want to cooperate |
Beta Was this translation helpful? Give feedback.
10 replies
-
|
I should had caught that too! Well the good news is that it does submit now and runs. But the bad news is that it fails. Still, one step closer I suppose. It fails in a different way than before. |
Beta Was this translation helpful? Give feedback.
-
|
CThis looks easy... this type of error happens when some output stream is trying to overwrite another. Could you please share the stdout of this command in case_scripts under this case? my guess is that you're duplicating the same file... |
Beta Was this translation helpful? Give feedback.
-
|
That's good news. I did add that inside case_build() but it doesn't seem to produce an output. However, I commented out the following, approximately at line 297:
Now I get a "completed" code in the slurm email after it runs. Yay. Is this appropriate or will it break something else? |
Beta Was this translation helpful? Give feedback.
-
|
That's fine, it will just mean you're not outputting anything as part of the simulation from the EAMxx side. If you're interested in profiling the IO layer, then you can try to insert it back carefully. But, I think the IO layer isn't as interesting in terms of profiling. So I would recommend ignoring it at least to start. |
Beta Was this translation helpful? Give feedback.
Answer selected by
mahf708
-
|
Got it. But regardless of profiling, if I don't output an output file, I cannot later visualize the simulation results, right? |
Beta Was this translation helpful? Give feedback.
-
|
It should be fixable. Could you move this line -./atmchange output_yaml_files+=/global/homes/m/mihelog/ThrustE/mihelog/testruns/scream_3hr_average_output.yaml
+./atmchange output_yaml_files=/global/homes/m/mihelog/ThrustE/mihelog/testruns/scream_3hr_average_output.yamlTo the runtime_options() function of your script? Also, note the minor detail: I changed |
Beta Was this translation helpful? Give feedback.
-
|
The output call shouldn't in in the build section. There are some complex details about how the build happens, so it's likely something gets done twice (and it would be wasteful to try to debug that). Let's just do the customary thing of adding that line to runtime_opts for ezpz sol |
Beta Was this translation helpful? Give feedback.
-
|
This seemed to do it. Thanks again so much! I have a working copy now I can tweak for profiling. |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone, Since I have the author of the scripts here I was wondering if I can get help on something different. I'm trying to add a constraint in the slurm flags. The equivalent in a sbatch script would be a -C "gpu&hbm80g" (these are higher-memory GPUs in NERSC). I can't figure out where to add this constraint in the collection of scripts. Any ideas? Thank you! |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
Have you tried |
Beta Was this translation helpful? Give feedback.
-
|
That's what I was looking for! Can you clarify though what directory I should run it in? I tried several combinations and I always get: |
Beta Was this translation helpful? Give feedback.
-
|
You should be in the case directory. I am assuming you are using a run script, in which case you prob have a var storing the case root, like CASE_ROOT in this script? In any case, it should be wherever |
Beta Was this translation helpful? Give feedback.
-
|
There are several ways to do it. This seems to work: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Thank you both! I figured out xmlchange but that didn't work for me since I had a layer of scripts that generated more scripts... and yada yada. But the environment variable and case.submit led me down the right path and I figured out where to append to the parameters string. Thank you again much! |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Glad you figured it out. If there's nothing more, go ahead and close the discussion, please. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
That's fine, it will just mean you're not outputting anything as part of the simulation from the EAMxx side. If you're interested in profiling the IO layer, then you can try to insert it back carefully. But, I think the IO layer isn't as interesting in terms of profiling. So I would recommend ignoring it at least to start.