Create a Python venv for the nuopc-app-prototypes

[rsdunlapiv]

pyyaml needs to be available for the ESMX_* prototypes

[theurich]

@rocky and @rokuingh - After the call I tested out the procedure manually on Discover with ESMX. Using the default python3 on the system, which is 3.4, I cannot load pyYaml. It seems to require version > 3.4!
So I ended up first with module load python/GEOSpyD/Min4.11.0_py3.9, and that then worked, but then in the commands could not use python3, but had to use python instead. So the module brings in a new path where there is a python (version 3.9), but does not have a python3 there!
Bottom line, we do need to provision for this in the test scripts!

[billsacks]

My understanding is that that's a system issue that the system maintainers should resolve: My understanding from https://peps.python.org/pep-0394/ is that python installations should install a python3 executable. (In CESM, we recently went through the exercise of changing all of our python shebang lines to #!/usr/bin/env python3, relying on the existence of a python3 executable based on that PEP.)

[theurich]

Okay, that was a false alarm from my side. I am not sure what I thought I was seeing before, but all looks good now. I.e. after module load python/GEOSpyD/Min4.11.0_py3.9 on Discover, I do have python3 pointing to the 3.9 version.
I think what my have happened before was that I did not distclean the ESMX_ proto, and therefore the CMake build directory hung around, and had the other python3 cached.
Main thing, it is working correctly with python3 after the module load - as it should. Sorry about the confusion.

[rsdunlapiv]

@theurich to facilitate creating the venv from the surround test scripts, can you please send the full list of commands you ran manually on Discover?

[theurich]

@rsdunlapiv , here is what I did:

module load python/GEOSpyD/Min4.11.0_py3.9
python3 -m venv ESMXenv
source ESMXenv/bin/activate.csh
pip list
pip install pyyaml

My default shell on Discover is TCSH, so that is why the *.csh flavor in the source line.

[theurich]

@rsdunlapiv - there is a new twist on Discover. I just noticed that when I load the python/GEOSpyD/Min4.11.0_py3.9 module, there is a nc-config in its PATH! Essentially this is overriding the nc-config that would be associated with the NetCDF module loaded. This is causing linking issues downstream.
I am testing whether just depending on the NetCDF that comes with python/GEOSpyD/Min4.11.0_py3.9 gives me a consistent environment... but I wonder if this might only work for Intel then, and not GNU. Honestly Discover looks like a big mess to me right now.

[theurich]

@rsdunlapiv - After some more testing, here is my final recipe on Discover for ESMX testing. This probably also applies to ESMPy testing on Discover:

1. Build ESMF as usual according to the combos specified in ./config/discover.yaml. Do NOT load module python/GEOSpyD/Min4.11.0_py3.9!
2. Use the python 3.9 executable with full path to set up the venv environment:

• /usr/local/other/python/GEOSpyD/4.11.0_py3.9/2022-04-28/bin/python3 -m venv ESMXenv
• source ESMXenv/bin/activate.csh
• pip install pyyaml

In my testing, the above procedure ensures there isn't anything funny going on with the NetCDF environment during compiling, loading, and running of ESMX. At the same time it does set up the Python environment so ESMX has what it needs.

[theurich]

For now we are getting away with just loading an appropriate python module for a specific combo with "extra_module". E.g. for Orion, I added intelpython3 like this:

extra_module: "cmake intelpython3"

With that, I did not have to go and implement the full procedure as used for ESMPy. In the future it might become necessary to run the NUOPC apps protos under a Python venv as outlined above. But for now we there is no platform where the simpler extra_module approach does not work.

I will keep this ticket open as a reminder in case there is a problem with it in the future.