Implementation and managing the various software requisites in an enterprise production environment

[couckearthur]

I want to have an open discussion on how to implement and manage the various assets needed for Ansible for z/OS,
especially the question about the versions. It goes a bit broader than z/OS Ansible but this looked the best place to ask it...

I will start the discussion by explaining how we implemented our z/OS Python and ZOAU setup. Afterwards I will formulate some questions we are discussing about internally.

z/OS Python

• We install our z/OS Python through ShopZ. as IBM states it provides easy update management. It is recommended for production deployments. Both licenses and S&S are available through Shopz

• We have the IBM AI Toolkit mirrored in our internal Artificatory, if one wants to install a package it is retrieved from there.

• We enforce virtual environments by default. A little in house written bash script is used to interactively create a venv and save it in pre set location. Also for activating we have a bash tool
  - the venvs are enforced to have users repsonsible for their own packages. We don't want conflicts about versions between the users
  - in this way we are able to scan scan all venvs packages automatically when we receive a vulnerability report from IBM, and notify the maintainer he has to update

• We have three symbolic links that points to IBM Python versions. e.g.
  current -> 3.11
  previous -> 3.10
  new -> 3.12 (to test)

• the current is set in the PATH in the general profile. So all users on mainframe without additional config are using this version
  After an IPL we notify all z/OS Python users by email to notify that they can start testing the new Python version (on the new symbolic). With another bash utility they can create 'test venv' in their home directory to start testing with the new (but not general active) Python version.

After the key users confirm, we change the current symbolic and the new version becomes the general active one.

• after the new version is installed all users have to reinstall their packages in their venv, this is because in their venv the python is a symbolic to the current (other version). After they done this they need to pax and ftp their packages library to the higher environments to be ready to use.

ZOAU

• here we keep two versions. In the past a new version replaced the old but since 1.3 it is now possible to have them next to each other

• Here we don't set the version in the general profile. We rather don't do it because ZOAU has additional security set and not all users are allowed to use it. But we do provide two 'set scripts' to set the env variables that enables to use ZOAU (if you're allowed)

  • the scripts have the version name in it but there are also two symbolic links to be used, current and previous

The questions

Can we enforce users to upgrade their Ansible collections?

In my view, if IBM has put a certain z/OS Python version in EOS we should have it rolled-off on our systems. That means that at a certain point all Ansible users have to install a new Ansible Collection, and will possibly have to make code changes and retest. I don't think it is a good idea to keep version of z/OS Python or ZOAU forever

maybe on distributed system this is a normal way to go? I don't know, but on mainframe it would be a first timer because we always had the backwards compatibility... as an infrastructure team we never 'broke code' with our updates, now it is possible

Python Version | GA | EOM (marketing) | EOS
3.8.x | 26-Jun-2020 | 06-Sep-2021 | 30-Sep-2022
3.9.x | 15-Jan-2021 | 06-Mar-2023 | 30-Apr-2024
3.10.x | 10-Dec-2021 | 04-Sep-2023 | 30-Sep-2024
3.11.x | 16-Dec-2022 | n/a | 30-Apr-2025
3.12.x | 17-Nov-2023 | n/a | n/a

Can we enforce users to upgrade their venvs?

This is a sensitive one. In a mainframe way of thinking backwards compatibility is holy. Teams are shiviring if we tell them that they have to test and update their code regulary, that is, every time we update the Python version (or their code could possibly break).
In my view if you want to use a non-native mainframe language like Python you also have to bear the consequences. That is updating, upgrading your code together with constantly evolving language and packages you wrote it in. You know it before you start using it.

So, it would help if some people of the community could read and comment on this. We're still searching our way with new things in our castle. I don't want to make certain assets management mistake which I will regret later on when Python, Ansible, ZOAU is more extensively used by different teams in the company.

[ddimatos]

@couckearthur - this is a great topic, albeit not easy over text to discuss so bare with us if we don't interpret everything as it was intended to be. I will respond first to the question, then follow that by a mix of what we have done, plan to do, and some suggestions.

Can we enforce users to upgrade their Ansible collections?
When you say we, I am going to interpret this as if both you and us as the offering if we can enforce this.

Strictly speaking from a mainframes visibility, no (noticed I stated mainframes visibility). Simply because Ansible runs momentarily on z/OS, it does not install on z/OS such that a configuration is in place where you could dictate usage. You are correct in that as dependencies ZOAU and/or Python reach EOS the ansible collection on the controller will need to be updated. What will happen, is when Ansible runs on z/OS with unsupported combinations (products that have reached EOS) their playbook will fail with an explanation. At the point, if they are savvy enough, they will realize what happened else they will follow the organizations process on how to obtain service.

However, if you are using Ansible Automation Platform (AAP) , I believe in APP 2.0 Execution Environments (EE) was introduced; if so, your organization could force the use of execution environments with frozen collection versions. Execution environments are basically containerized Ansible with the collections all in one, those EEs will be how AAP connects to z/OS every time a playbook is run, users of APP will not be able to alter this behavior. When the time comes for a collection and dependency combination change, you can build a new EE , and that will be used by AAP, users will not be aware (of course unless interfaces or options have changed).

EEs are available to non AAP users as well, the containers can be built and hosted, and they can run the organizational defined EE this way the collection, distributed python, any python packages that normally would be required on the controller would be frozen.

All our collections are EE ready to be built, we have put the meta data into the collection.

What can this offering do? Well, dependency checks is something we want to get to, where we would check for compatibility and the reason its not there yet is because we want to carefully think about about this and not roll something out because this will be an incurred cost each time a module connects and we want to avoid overly taxing the modules execution. Long ago we thought about this and embedded some meta data you are welcome to use, we have not used it in this collection but it is used in our samples repository. You can find it here, but this is not a public interface and subject to change when we decide to use it.

You might ask well how could I use this ibm_zos_core_meta.yml, well its a bit out of the box thinking but what if, you regularly had z/OS call out to this repository (or your private clone of it) or monitor it for changes and in that call out you consume the contents of this file and see there is a new version and its dependencies have been changed (below) indicating that now Python 3.9 is no longer supported and you then email everyone or take a more aggressive action and countdown till access is limited. The problem with this is its only a partial solution because you don't know what version everyone is using since collections are not left behind on z/OS, unless of course you required every playbook to write to a location the collections version and maybe user name so you had some sort of on going accounting.

        name: "IBM Open Enterprise SDK for Python"
        version: ">=3.10"

Can we enforce users to upgrade their venvs?
I am not sure under what context, do you want to force venv upgrades as a result of a CVE patch? Do you want to force it as a result of EOS of say Python? Or do you just want to regularly force them to update their venv's based on the release cadence of ansible collections? I guess I don't know what condition would warrant this enforcement so I am not 100% sure how to respond but I can include some of my thoughts.

Similar to above, how would you know when, well there are 2 ideas above, each playbook make an entry of its version and username and you read that as your source of truth to force a venv upgrade. The second was to read the collections meta data and react to that, also mentioned above.

I would think if you carefully constructed the venv, where the environment variable paths don't change they would not be aware (unless something broke) , the venv will give you a consistent python path , so that is no issue, it won't change from venv to venv. For zoau you can perform a pip install if they have permission into their venv with pip's --target option and then the environment vars would have to use PYTHONPATH to point to that location, and you can defined what that path looks like so it never changes, what i mean is you don't have anything in the path that is subject to change, eg /u/usr/admin/zoau/3.12/ , instead /u/usr/admin/zoau/python.

There is one more tool that could help, there is a concept of adding custom facts to be consumed an ansible playbook which users could use to be cognizent of change about to occur, you could have a key:value pair where the key might be, "upgrade:recommended" or "upgrade:required" , some other keys with dates and such. Problem with this is its on the users playbook to do this check. Once you create a /etc/ansible/facts.d directory, you can define facts as JSON, an INI or even a executable script, the playbook will receive those and act accordingly.

In some ways we have a similar set up but because we are development we are not overly concerned about being forced to upgrade, but we do follow similar processes. In your case you are sharing a system, with us, we have over 40 zVMs we rebuild weekly (you can think of these are our venv) , we mount shared DASD that has all the pythons and zoau's and then let the developers switch between them as can our pipelines. On the controller we use venv's , we have a tool that aids with that, it creates them and manages them, this might be something you can use on z/OS, it might take some tweaking. One of the things the tool does after it mounts the shared DASD it writes a interactive profile allowing users to choose which every ZOAU and Python combination they need, this is done for support purposes.

Overall, you are on the right path, your suggests are inline with what I would have done, while I don't know all the details of the profile, I would probably have the script let users choose where to create the venv, the script would write this out to a secure file with permission of 600 and then you can use that to find all the venvs.

We have an upcoming guild call next week, this could also be a great place to discuss this, the Ansible guild we host is open to everyone, quite a few customers join as well.

I am open to a call if it will help, only because I feel this should probably been something we prescribed long ago to help our users, where I am thinking this is still something we should explore and advise one.

[ddimatos]

I forgot to share the script I mentioned that we use, it starts here at this tool, then all the scripts can be found here.

[couckearthur]

Hello Demetri,

Thank you for the detailed explanation. It is interesting to read how you do it on your development environment.
What you mentioned about the zVM made me think about zOSCP. I'm checking with IBM, but if there would be (in the future) native containers with z/OS Python and ZOAU it would solve the version quest (and conflicts) for different users. The version management would move from the infrastructure group to the end user, owner of the container.

[ddimatos]

@couckearthur , having containers images available with the Ansible Z collections + python + zoau is a nice idea to ease dependency requirments, I will bring this up with some of the other teams I work with who are building containers for their offerings.

I am not sure if you are familiar with Execution Environments, while they don't address the managed node dependencies, you can use them to to address controller dependencies. For example, if you are building a solution on a test system, if you use EEs as the base for Ansible, you can then use the same frozen image for production.

[ddimatos]

Feel free to bring up other questions, if you feel you want to discuss this further I would be open to a call or during our monthy guild call , etc, for now it seems we can close this but feel free to reopen as well.