Can't create file / delete non expired

[gpow81]

Hi,
This time I am having problems with creating new datasets and copying files tot he z/OS. I am sure that this is just problem with me not understanding z/OS enough but I was hopping someone could point me into right direction.
Whenever I run a simple playbook which involves creating a new file, for example:

• name: Copy a local file to a sequential data set
  zos_copy:
  src: /tmp/sample
  dest: 'MYUSER.SEQ.DDATA.SET'
  register: result

First of all I am forced to accept a bunch of temporary data sets via SD, log or the playbook just waits forever:

*IEC507D E 500D,PUBLC1,GPOWIED8,STEP1,GPOWIED.P3554789.T0752483.C0000000
*55 IEC507D REPLY 'U'-USE OR 'M'-UNLOAD                                 

and then after few of these, the playbook errors out saying that it can't define the file

tderr_lines": [
        "cp: FSUM6258 cannot open file \"//'GPOWIED.SEQ.DDATA.SET'\": EDC5061I An error occurred when attempting to define a file to the system."

At the same time z/OS is complaining about not being able to delete an unexpired dataset which is probably the main issue:

 STEP1,PUBLC1,GPOWIED.P3554789.T0752483.C0000000                         
 IGD17057I DELETE FAILED, UNEXPIRED PURGE DATE ON VOLUME PUBLC1, 369     
 FOR DATA SET GPOWIED.P3554789.T0329525.C0000001                         
 IEC614I SCRATCH FAILED - RC 008, DIAGNOSTIC INFORMATION IS (040B004B),  
 370                                                                     

I am able to delete and purge these files manually afterwards so I am surprised that ansible can't do it. Am I missing some extra settings here or permissions are incorrect?
BTW, what are these temporary files for?

thank you!

[ddimatos]

Hi @gpow81 - we can answer some of your questions with more accuracy but I am not sure which version of the ibm_zos_core collection you are using.

It looks like you trying to create a data set with HLQ MYUSER, I can't know how you system is configured to allow HLQ, but you can try a few things.

Try:

1. using what appears to be your user as the HLQ

    - name: Copy a local file to a sequential data set
      zos_copy:
        src: /tmp/sample
        dest: 'GPOWIED.SEQ.DDATA.SET'
      register: result

2. Try without but you system may add it then resulting in the file not found

    - name: Copy a local file to a sequential data set
      zos_copy:
        src: /tmp/sample
        dest: 'SEQ.DDATA.SET'
      register: result

3. Try overriding the HLQ, using an EXPORT, we are in our next release including the override [ zos_*_modules ] Update modules with new ZOAU Temp HLQ feature - Nazare 6164 tmphlq #341 as an option to the module instead of an export, to use it as an export add it to your environment vars like so (notice TMPHLQ: "GPOWIED" ):

environment_vars:
  _BPXK_AUTOCVT: "ON"
  ZOAU_HOME: "{{ ZOAU }}"
  JAVA_HOME: "{{ JAVA }}"
  PYTHONPATH: "{{ ZOAU }}/lib"
  LIBPATH: "{{ ZOAU }}/lib:{{ PYZ }}/lib:/lib:/usr/lib:."
  PATH: "{{ ZOAU }}/bin:{{ PYZ }}/bin:/bin:/var/bin:{{ JAVA }}/bin"
  _CEE_RUNOPTS: "FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
  _TAG_REDIR_ERR: "txt"
  _TAG_REDIR_IN: "txt"
  _TAG_REDIR_OUT: "txt"
  LANG: "C"
  TMPHLQ: "GPOWIED"

As for all the temporary data sets, its not usually the users option to control behaviors, the reason they are used depends on the code path, if a z/OS program is used in the modules execution, they might require Data sets as input/output (SYSIN, etc) thus we create them and manage their lifecycle. In your case a file is being copied from the controller to z/OS , Ansible runs in USS, we stage it temporarily while we handle encoding conversions before we copy it into a data set, I would have to analyze the exact code path to provide the correct answer but that is a summary.

We have a few current work items happening, zos_copy 1.4.0-beta.2 is in the works and its improving any temporary file/data set usage , in another work item we are monitoring all temporary allocations and ensuring that even during a module crash we don't leave any behind.

[gpow81]

Thank you for the answer. I apologize but I just added unnecessary confusion by trying to sanitize logs and replacing my actual userid with a generic "myuser". So I am not trying to use a different HLQ, I am just bad at copy and paste.
ibm_zos_core=1.3.5

For some reason, some of my test playbooks are working today. I don't think I am doing anything different. Also z/OS admin hasn't done anything. But I can:

• create a dataset
• I can copy a file from my control machine to the remote z/OS, which is a big step, by using a simple :

    - name: Copy a local file to a sequential data set
      zos_copy:
        src: /tmp/sample
        dest: 'GPOWIED.SEQ.DATA.SET'      (  'SEQ.DATA.SET' works as well) 
      register: result

Although the sample copy_fetch_data_set.yml is still having problems. It actually is getting one step further (past creating an empty PDS). This time it fails on "Copy local file to the created PDS member"

fatal: [zos_host]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "dest": "GPOWIED.P0001206.T0961987.C0000000(HELLO)",
            "encoding": {
                "from": "ISO8859-1",
                "to": "IBM-1047"
            },
            "src": "/root/ans_zos_collections1/z_ansible_collections_samples/zos_concepts/data_transfer/copy_fetch_data_set/files/HELLO.jcl"
        }
    },
    "msg": "Failure during execution of mvscmd; Return code: 8; stdout: 1READY                                                                                                                   \n   LISTDS 'GPOWIED.P0001206.T0961987.C0000000'                                                                           \n GPOWIED.P0001206.T0961987.C0000000                                                                                      \n --RECFM-LRECL-BLKSIZE-DSORG                                                                                             \n   FBA   100   27900   PO                                                                                                \n --VOLUMES--                                                                                                             \n   PUBLC1                                                                                                                \n READY                                                                                                                   \n END                                                                                                                     \n; stderr: BGYSC0220E Error deleting temporary dataset SYSTSIN=GPOWIED.P0001229.T0914989.C0000000.\nBGYSC0220E Error deleting temporary dataset SYSTSPRT=GPOWIED.P0001229.T0227586.C0000001.\n",
    "rc": null,
    "stderr": null,
    "stderr_lines": null,
    "stdout": null,
    "stdout_lines": null

The playbook left behind 5 temporary datasets, in one of these I can see this:

IDCAMS  SYSTEM SERVICES                                           TIME:
                                                                       
 LISTCAT ENTRIES('GPOWIED.P0001206.T0961987.C0000000')                 
IDC3012I ENTRY GPOWIED.P0001206.T0961987.C0000000 NOT FOUND            
IDC3009I ** VSAM CATALOG RETURN CODE IS 8 - REASON CODE IS IGG0CLEG-42 
IDC1566I ** GPOWIED.P0001206.T0961987.C0000000 NOT LISTED              

Which is strange because I can see it in 3.4 or by running a tso listcat command.
the next temporary file says:

READY                                          
  LISTDS 'GPOWIED.P0001206.T0961987.C0000000'  
GPOWIED.P0001206.T0961987.C0000000             
--RECFM-LRECL-BLKSIZE-DSORG                    
  FBA   100   27900   PO                       
--VOLUMES--                                    
  PUBLC1                                       
READY                                          
END                                            

On another note, is that normal that I need to confirm each temporary file creation in sd->log by replying /NNu ? Is there a way to avoid that.

thank you! I appreciate your patience.

[ddimatos]

No problem. I see the playbook you are running, I have not run it on an zoau 1.2.0.1 and python 3.10 to see how it behaves, but I can try it when i have some free time this week.

As far as the error, I will need to look into the zos_copy module source but i recall one of the prior behaviors of the module was to not check if a data set is empty reuse it, it would try to create a new one using the dest options (model it) and it looks like that design issue. The latest coming out in a couple of weeks 1.4.0-beta.2 takes into account such things even when you don't specify a record record length it will figure it out, anyhow the error I see is:

 stderr: 
  BGYSC0220E Error deleting temporary dataset SYSTSIN=list.
  BGYSC0220E Error deleting temporary dataset SYSTSPRT=GPOWIED.P0001229.T0227586.C0000001."

Which makes me think it first created the new DS GPOWIED.P0001229.T0227586.C0000001, was planning a rename after it deleted GPOWIED.P0001206.T0961987.C0000000 but failed on delete. I don't know why it failed to delete, not sure if that is a system issue where you have to confirm deletes as well.

So, let me follow up after I try this and capture some of the logs.

As for "is that normal that I need to confirm each temporary file creation in sd->log by replying /NNu " that is not normal for us, nor have I ever heard a customer share this, I will ask some of the customers and check with others on this.

Let me try this, i should have this combination on my system to try out,

[ddimatos]

Its been a while, since our last discussion, version 1.4.0 ibm_zos_core has released and zos_copy was completely rewritten, I was not able to recreate this.

We can continue to try to diagnose this but would ask it to be on version 1.4.0 ibm_zos_core. I hope the issues are resolved.

[gpow81]

Thank you, my z/os account definitely had some permissions problems and it was writing temporary files to a wrong volume.
After fixing that, I was able to move further. I had to move to another project since then but eventually I will circle back to it.
Thank you for all the help. The issue is resolved

[ddimatos]

@gpow81 , thank you for the reply even though time has passed. We are working towards reducing temporary files, feel free to reach out us anytime. Happy automating :).