z/OS core - Job submit failed

[fabiobncbb]

I'm using z/OS core 1.4.0 to submit jobs and encountering the following situation:

• When the sysout is small or the job abends with return-code 12, the automation ends correctly and I manage to handle the sysout.

• When the sysout is large, the automation return fails with the error in the image below, but in the mainframe it runs normally.

Obs.:I already tried with return output false and true
Obs.: I already increased the size of the user's TSO session on the mainframe, the user's /home on the control node but I can't identify exactly what the problem is.

[ddimatos]

Hello @fabiobncbb
I'll look closer at this shortly, I have a suspicion this is related to an issue a user raised in an email to me that I was not able to reproduce and later when I engaged them they were not able to allocate time to further this along which left the analysis incomplete.

I would be happy to help and it just happens we might still have a customized build you could try that would answer if this issue is yours as well , here. If I recall you might have challenges installing updates, I'll review on Monday and reach out you.

[ddimatos]

@fabiobncbb by any chance did you noticed any system log entries that look like:

INSUFFICIENT STORAGE
NOT ENOUGH STORAGE FOR BUFFER

[fabiobncbb]

Unfortunately no, in the z/OS log messages this message does not appear.

Today, scanning the log messages of both executions (successfully and unsuccessfully), I identified a SEC6 abend in the execution where I cannot capture the sysout, but I could not advance in this analysis.

Esta é a única diferença que encontrei entre as duas execuções até o momento.

[ddimatos]

So I did trace it down to the code which threw the exception which then was propagated by our code instead of capturing it which would not change anything.

From what I can see without a recreate is that after our REXX code ran which collected the job log, the Ansible engine code started reading stdout and and appending it to a variable which for some reason appears to run out of storage/heap. The issue I noted in the post #80379 is not the same issue, although its in the same ansible api.

I have one idea, this would not work for issue #80379 but might work for this, and that is to force the stdout and stderr streams to be unbuffered by using environment variable PYTHONUNBUFFERED.

You should be able to do this by passing it to your environment vars like so PYTHONUNBUFFERED: 1.

environment_vars:
  _BPXK_AUTOCVT: "ON"
  ZOAU_HOME: "{{ ZOAU }}"
  JAVA_HOME: "{{ JAVA }}"
  PYTHONPATH: "{{ ZOAU }}/lib"
  LIBPATH: "{{ ZOAU }}/lib:{{ PYZ }}/lib:/lib:/usr/lib:."
  PATH: "{{ ZOAU }}/bin:{{ PYZ }}/bin:/bin:/var/bin:{{ JAVA }}/bin"
  _CEE_RUNOPTS: "FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
  _TAG_REDIR_ERR: "txt"
  _TAG_REDIR_IN: "txt"
  _TAG_REDIR_OUT: "txt"
  LANG: "C"
  PYTHONUNBUFFERED: 1

Try this and see if it helps, if you have any stats/info on the joblog size in terms of kb or lines we might be able to reproduce it.

[ddimatos]

@fabiobncbb - I did not have time to write the playbook till today, but included in this response is a playbook that will query some heap stats. What I am looking for is why a variable in python did not have enough heap which I was hoping PYTHONUNBUFFERED might help. I suspect your OMVS Segment MEMLIMIT might be limiting the process/user ulimit setting for above the bar, or the setting for above the bar is to low for what you need.

Here is the playbook, choose what you feel comfortable sharing, but at minimum I need these tags:

#   - "shell_info" - display the shell
#   - "ulimit_info" - display all ulimit settings

which can be run like so: ansible-playbook -i inventory zos_diagnostic.yml --tags "shell_info,ulimit_info"

the others will be helpful, the playbook has the instructions written in side.

# ##############################################################################
# Usage:
#   ansible-playbook -i <inventory> <playbook> --tags "....."
#   eg `ansible-playbook -i inventory zos_diagnostic.yml --tags "shell_info,ulimit_info"`
#
# tags are all documented below in the playbook:
#   - "shell_info" - display the shell
#   - "ulimit_info" - display all ulimit settings
#   - "lu_info" - display information about the omvs segement running this playbook
#   - "sys_limit_info" - display the system limits in place
#   - "sys_options_info" - display the styem options
# ##############################################################################

---
- hosts: zvm
  gather_facts: false
  environment: "{{ environment_vars }}"
  collections:
    - ibm.ibm_zos_core

  tasks:
    # ------------------------------------------------------------------------------
    # Display user
    # ------------------------------------------------------------------------------
    - debug:
         msg: User executing playbook is "{{ ansible_user }}"

    # ------------------------------------------------------------------------------
    # Get users shell information
    # ------------------------------------------------------------------------------
    - name: Get shell information
      shell:
        cmd: "echo $0"
      register: result
      tags: shell_info

    - name: Result shell info
      debug:
         msg: "{{result}}"
      tags: shell_info

    # ------------------------------------------------------------------------------
    # Get ulimit information
    # ------------------------------------------------------------------------------
    - name: Get ulimit information
      shell:
        cmd: "ulimit -a"
      register: result
      tags: ulimit_info

    - name: Result ulimit
      debug:
         msg: "{{result}}"
      tags: ulimit_info

    # ------------------------------------------------------------------------------
    # Get Segement information
    # ------------------------------------------------------------------------------
    - name: List user {{ ansible_user }} OMVS
      zos_tso_command:
        commands: "lu {{ ansible_user }} omvs"
      register: result
      tags: lu_info

    - name: Result list user {{ ansible_user }} OMVS
      debug:
         msg: "{{result}}"
      tags: lu_info

    - name: Display system limits
      zos_operator:
        cmd: 'D OMVS,LIMITS'
      register: result
      tags: sys_limit_info

    - name: Result display system limits
      debug:
         msg: "{{result}}"
      tags: sys_limit_info

    - name: Display system options
      zos_operator:
        cmd: D OMVS,OPTIONS
      register: result
      tags: sys_options_info

    - name: Result display system options
      debug:
         msg: "{{result}}"
      tags: sys_options_info

    # --------------------------------------------------------------------------
    # Ulimit Notes:
    # --------------------------------------------------------------------------
    # core file    		  - The core file size allowed for when a program terminates
    #                     unexpectedly
    # cpu time          - The maximum amount time in seconds the CPU is  allowed
    #                     for a process
    # data size         - The maximum size of a processes data segement, in other
    #                     words if the limit is exceeded mallac would fail.
    # file size         - The maxium file size a process can create
    # stack size        - The maximum size (space) of the stack for a process and
    #                     thread.
    # file descriptors  - The maximum number of open file descriptors allowed for a
    #                     process
    # address space     - The maximum address space/size for the process, controls
    #                     the maximum number of threads and processes.
    # memory above bar	- The storage a process is allowed to have allocated and
    #                     unhidden above the 2 gigabyte bar. "The amount of storage
    #                     that `ulimit -M` displays does not necessarily reflect the
    #                     MEMLIMIT setting found in the user's RACF OMVS segment.
    #                     The value displayed will depend on how the user entered
    #                     the OMVS shell and whether a change of identity was
    #                     performed."
    # --------------------------------------------------------------------------

    # --------------------------------------------------------------------------
    # RACF
    # --------------------------------------------------------------------------
    # MMAPAREAMAX       - The ser's z/OS UNIX maximum memory map size
    # RACF MEMLIMIT 		- The users z/OS UNIX non-shared memory size
    # --------------------------------------------------------------------------

    # --------------------------------------------------------------------------
    # Links:
    # --------------------------------------------------------------------------
    # For info on umlimit:
    #   	https://www.ibm.com/docs/en/zos/2.4.0?topic=descriptions-ulimit-set-process-limits
    # RACF Virtual storage capacity
    # 	https://www.ibm.com/docs/en/zos/2.4.0?topic=resources-virtual-storage-capacity
    # RACF Profile Options
    #   https://www.ibm.com/docs/en/zos/2.1.0?topic=profiles-omvs-segment-in-user
    # Process limits:
    #   https://www.ibm.com/docs/en/zos/2.4.0?topic=bpxprmxx-displaying-process-limits
    # ------------------------------------------------------------------------------

[fabiobncbb]

After some tests, we detected that the unix/tso MEMLIMIT environment variable, in z/OS, was limiting the execution to 2GB and the operation was being canceled and generating the memory error.
After increasing this value to the maximum allowed in our environment (24GB) the problem was solved.