View All / Ansible, ZOA Utilities and ZIIPy Python

Ansible, ZOA Utilities and ZIIPy Python

James Gill

November 3, 2023

Get in touch

Introduction

A little while back, we started working with Python, the Z Open Automation (ZOA) Utilities and Ansible to get some neat automation examples running on z/OS. There are lots of things that we like about Ansible – its fact gathering, stateful approach and large and enthusiastic development community make it incredibly useful for reliable deployment automation.

Since IBM has become involved (cf z/OS Core, Hardware Management Console, z/OSMF, CICS, IMS, System Automation plus lots of storage goodness), the z/OS platform has enjoyed some DevOps focus. New releases of the Ansible collections appear on a regular basis and the scope and reach is ever expanding.

We thought it would be worth coming back and having another look – especially in light of the recent addition of ZIIP support to Python (IBM Open Enterprise SDK for Python).

Getting Setup

Python

We had a look at the ZIIP enabled Python from IBM (v3r11m0) just after it was released, but we thought it was worth bringing everything back up to date. The current (at the time of writing) version available as a pax download is 3.11.5, and can be found here:

https://www.ibm.com/products/open-enterprise-python-zos

The process was made slightly more exciting than we were expecting, as Microsoft Defender flagged the cloudy source of the pax file as a source of scamming content!

Having ignored this and pushed the file to z/OS, we unpacked it in USS with:

pax -r -ppAx -f HAMB3B0.nonsmpe.pax.Z

We re-deployed this as a refresh in the current path:

usr/lpp/IBM/cyp/v3r11/pyz

ZOA Utilities

This software has also been updating continuously since we first looked at it in 2021 (v1r0m3) and with the current version (v1r2m5) including updates to support more fact acquisition around IPL and storage activities. The pax version can be downloaded directly from IBM’s GitHub repos – see the Enterprise DevOps section:

https://ibm.github.io/mainframe-downloads/downloads.html

Having pulled the pax file and delivered it to z/OS, we unpacked it in USS with:

pax -p e -rf zoau-1.2.5.0.z

Having unpacked this and deployed this into the current path:

/usr/lpp/IBM/zoautil

We then have to configure this for the Python API. This is done using the Python package management utility (pip3):

cd /usr/lpp/IBM/zoautil
pip3 install zoautil_py-1.2.5.0-cp311-none-any.whl

The “cp311” part refers to the runtime Python version. Others supported in the delivery are:

Python 3.9 (cp39)
Python 3.10 (cp310)

Because our version of ZOAU was built against Python 3.9, it will look for libpython3.9.so – which won’t be available with Python 3.11. To overcome this, we created a convenient link:

Remount the Python ZFS in read / write mode:
- umount filesystem(‘TOOL.PYTHON.V3R11.ZFS’) remount(RDWR)
Switch to UID(0)
- su –
Create the link:
- cd /usr/lpp/IBM/cyp/v3r11/pyz/lib
- ln -s libpython3.11.so libpython3.9.so

If you skip this, you’ll get warnings about libpython3.9.so being unavailable.

Ansible z/OS Core Collection

Like the ZOAU software, the z/OS ansible collection has been updating very regularly. When we were first looking at this in 2021, it was version 1.2.1, and has recently advanced to 1.7.0 (which we’re looking at here), and we just saw an email about 1.8.0-beta.

We deployed the current (1.7.0) onto an Ubuntu Linux environment (under my personal user) using:

ansible-galaxy collection install ibm.ibm_zos_core --upgrade

Getting Ready to Run

We need one or two things configured to be able to run Ansible against z/OS:

Password-less ssh connectivity
Ansible inventory to reference z/OS
Environment variables/configuration

For reference – and to help these examples make sense:

Linux layer where we run the Ansible playbooks is Ubuntu
- User = james
z/OS layer is a zPDT (emulated mainframe hardware) running z/OS 2.5:
- IP address = 10.1.1.4
- SSH port = 6552
- Userid = IBMUSER

To sort each of the configuration items out in turn:

Password-less ssh connectivity:
- On Ubuntu with my user, generate the ssh key set:
  - ssh-keygen -t ecdsa
- Copy the resulting public key to IBMUSER on z/OS, so that we use IBMUSER for the ssh connection:
  - ssh-copy-id -p 65522 ibmuser@10.1.1.4
- Test that the configuration works by ssh-ing onto z/OS as IBMUSER and checking that we don’t get prompted for a password:
  - ssh -p 65522 ibmuser@10.1.1.4
- Create the Ansible inventory:
  - - On Ubuntu with my user, create a new path off my HOME
      - mkdir ansible
      - cd ansible
      - And create the inventory file (zpdt.yml):

source_systems:
hosts:
zpdt:
ansible_communication: ssh ansible_host: 10.1.1.4 ansible_port: 65522 ansible_user: ibmuser ansible_python_interpreter: /usr/lpp/IBM/cyp/v3r11/pyz/bin/python3

Create the zPDT connection environment variables. Host variable sets live in a subdirectory called “host_vars”, so:
- On Ubuntu, with my user, create the host_vars path:
  - cd ansible
  - mkdir host_vars
  - cd host_vars
- Create the environment variables for our connection with the host called “zpdt” in the inventory (cf environment_vars in playbooks, below):
  - vi zpdt.yml

# Environment vars for zpdt PYZ: "/usr/lpp/IBM/cyp/v3r11/pyz"/ ZOAU: "/usr/lpp/IBM/zoautil" JAVA: "/usr/lpp/java/J8.0_64"

# environment_vars
environment_vars:
LANG: "C"
_BPXK_AUTOCVT: "ON"
ZOAU_HOME: "{{ ZOAU }}"
PYTHONHOME: "{{ PYZ }}"
JAVA_HOME: "{{ JAVA }}"
PYTHONPATH: "{{ ZOAU }}/lib"
LIBPATH: "{{ PYZ }}/lib:{{ ZOAU }}/lib:{{ JAVA }}/lib:/lib:/usr/lib"
PATH: "{{ ZOAU }}/bin:{{ PYZ }}/bin:{{ JAVA }}/bin:/bin:/usr/bin"
MANPATH: "{{ ZOAU }}/docs/%L"

Now we can create some playbooks to run from our ansible directory – as in the examples below.

What Do We Like?

The documentation is good and provides some examples of use as well. You can find it here:

https://ibm.github.io/z_ansible_collections_doc/ibm_zos_core/docs/ansible_content.html

zos_apf
This allows us to list and update the APF list, making changes that are dynamic only, or applied to PARMLIB PROG members as well. This example lists the current APF list:

---

- name: "Get the current APF list"
hosts: zpdt
collections:
- "ibm.ibm_zos_core"
gather_facts: false
environment: "{{ environment_vars }}"

tasks:

- name: "Start timestamp"
debug:
msg: "{{ lookup('pipe','date') }}"

- name: "Get the current APF list"
zos_apf:
operation: list
register: apflist

- name: "Report the current APF list"
debug:
msg: "APF datasets : {{ apflist }}"

- name: "End timestamp"
debug:
msg: "{{ lookup('pipe','date') }}"

e.g. (NB – we’ve removed some of the spaces from this output for readability / doc size):

james@fx507:~/ansible$ ansible-playbook -i zpdt.yml get_apf.yml
PLAY [Get the current APF list] *************************************************
TASK [Start timestamp] **********************************************************
ok: [zpdt] => {}
MSG:

Wed Nov 1 16:58:59 GMT 2023
TASK [Get the current APF list] *************************************************
ok: [zpdt]
TASK [Report the current APF list] **********************************************
ok: [zpdt] => {}
MSG:
APF datasets : {'changed': False, 'stderr': '', 'rc': 0, 'stdout': '[{"format": "DYNAMIC"}, {"header": ["S0W1 2023305 16:59:36.00 ISF031I CONSOLE IBMU0000 ACTIVATED", "S0W1 2023305 16:59:36.00 -D PROG,APF", "S0W1 2023305 16:59:36.00 CSV450I 16.59.36 PROG,APF DISPLAY 933", "FORMAT=DYNAMIC", "ENTRY VOLUME DSNAME"]}, {"vol":"D5RES1", "ds": "SYS1.LINKLIB"}, {"vol":"D5RES1", "ds": "SYS1.SVCLIB"}, {"vol":"D5RES1", "ds": "SYS1.SHASLNKE"}, {"vol":"D5RES1", "ds": "SYS1.SIEAMIGE"}, {"vol":"D5RES1", "ds":

{"vol":"D5PRD1", "ds": "FEUE00.SFEUAUTH"}, {"vol":"D5ZWE1", "ds":
"ZWE200.SZWEAUTH"}]'], 'stderr_lines': [], 'failed': False}
TASK [End timestamp] ************************************************************
ok: [zpdt] => {}
MSG:
Wed Nov 1 16:59:14 GMT 2023
PLAY RECAP **********************************************************************
zpdt : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

zos_archive
This creates an archive in a variety of different formats (tar, gz, zip, terse and xmit). We have a tool for reading xmit offloads of PDS datasets, so this is really useful when paired with the zos_fetch module. Quite useful with zos_unarchive as well!

E.g.

---

- name: "Get XMIT Archive of JCL"
hosts: zpdt
collections:
- "ibm.ibm_zos_core"
gather_facts: false
environment: "{{ environment_vars }}"

tasks:

- name: "Start timestamp"
debug:
msg: "{{ lookup('pipe','date') }}"

- name: "Create an XMIT Archive for GILLJ.SYSTEM.JCL"
zos_archive:
src:
- "GILLJ.SYSTEM.JCL"
format:
name: "xmit"
dest: "GILLJ.SYSTEM.JCL.XMIT"
force: true

- name: "Fetch the archive"
zos_fetch:
src: "GILLJ.SYSTEM.JCL.XMIT"
dest: "/tmp/"
is_binary: true
flat: true

- name: "End timestamp"
debug:
msg: "{{ lookup('pipe','date') }}"

zos_backup_restore
This is an interface to the ADRDSSU BACKUP and RESTORE commands. The resulting backups are tersed (cf AMATERSE) into fixed block datasets (LRECL=1024). This makes them very easy to move / copy (binary transfer) between LPARs. We use this functionality to create a maintenance package which we can then pus onto Nexus RM, and pull and unpack / restore on the target service.

E.g. Dump/Terse:

---

- name: "Pack CPP and JCL with LOADLIB"
hosts: zpdt
collections:
- "ibm.ibm_zos_core"
gather_facts: false
environment: "{{ environment_vars }}"</code

tasks:

- name: "Backup datasets (DUMP/Terse)"
zos_backup_restore:
operation: "backup"
data_sets:
include:
- "GILLJ.SYSTEM.JCL"
- "GILLJ.CPP"
- "GILLJ.LOADLIB"
backup_name: "GILLJ.CBU.TRS"
overwrite: true

And Unterse/Restore – to a different HLQ:

---

- name: "Unpack CPP and JCL with LOADLIB to Harry's HLQ"
hosts: zpdt
collections:
- "ibm.ibm_zos_core"
gather_facts: false
environment: "{{ environment_vars }}"

tasks:

- name: "Restore datasets (Unterse/RESTORE)"
zos_backup_restore:
operation: "restore"
backup_name: "GILLJ.CBU.TRS"
hlq: "POTTERH"
overwrite: true

zos_copy
Copy files – either inside z/OS or from the outside into z/OS and USS. Optionally manage codepage translations as well.

E.g. copy our XMIT file that we offloaded back to z/OS with a new name:

---

- name: "Copy XMIT file to z/OS"
hosts: zpdt
collections:
- "ibm.ibm_zos_core"
gather_facts: false
environment: "{{ environment_vars }}"

tasks:

- name: "Copy GILLJ.SYSTEM.JCL.XMIT to z/OS"
zos_copy:
src: /tmp/GILLJ.SYSTEM.JCL.XMIT
dest: "POTTERH.SYSTEM.JCL.XMIT"
is_binary: true
force: true

zos_encode
Convert a file or dataset from one encoding scheme to another. This can be done in place or write the result to a new file or dataset. Note that the module is fairly strict on how you refer to the codepages in use – the list of valid names/values can be found here. Note also that you don’t get any CR/LF conversion into records, so this is probably of more value working with USS files, than shifting data to MVS datasets.

E.g. copy a bit of Python from USS to a new sequential dataset:

---

- name: "Copy a USS file to a PS and convert from 1047 to 37"
hosts: zpdt
collections:
- "ibm.ibm_zos_core"
gather_facts: false
environment: "{{ environment_vars }}"

tasks:

- name: "(Re)Create target dataset"
zos_data_set:
name: POTTERH.CHKZOA.PY
state: present
replace: true
type: SEQ
space_type: TRK
space_primary: 1
space_secondary: 1
record_format: VB
record_length: 250
block_size: 27900

- name: "Copy and re-encode"
zos_encode:
src: /u/ibmuser/chkzoa.py
dest: POTTERH.CHKZOA.PY
encoding:
from: IBM-1047
to: IBM-037

zos_gather_facts
This module gathers configuration data from z/OS – e.g. the current IPL configuration (cf D IPLINFO), CEC info, system data or all of the above. The “sys” (system) option is useful for verifying the system and/or SMF name before performing other activities.

E.g. gather and report the current IPL:

---

- name: "Report current IPL"
hosts: zpdt
collections:
- "ibm.ibm_zos_core"
gather_facts: false
environment: "{{ environment_vars }}"

tasks:

- name: "Collecting IPL data"
zos_gather_facts:
gather_subset: ipl

- name: "What did we get?"
debug:
msg: "{{ ansible_facts }}"

What Don’t We Like?

There are two challenges with the ZOAU installation – one is the need to add a link for libpython3.9.so (see the ZOA Utilities installation notes, above), which wasn’t mentioned in the documentation (at the time of writing).

The second one is caused by extra data being returned by the ZOAU command “zoaversion” which confuses the ansible z/OS Core functions that use it to check whether we’re at or above the required level:

IBMUSER:/u/ibmuser: >zoaversion
2023/09/20 17:20:50 CUT V1.2.5.0 dfc0ed10 3938 PH54246 998 7544c1ea

And here, where ansible is attempting the “zos_gather_facts” for the “ipl” subset:

TASK [Collecting z/OS config]********************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ValueError: invalid literal for int() with base 10: '544c1ea‘

A little debugging later, it seems that the supplied build information file contains a bit more data than required. As delivered:

# Generated file. Do not edit! build: 2023/09/20 17:20:50 CUT V1.2.5.0 3938 git-commit: dfc0ed10 APAR: PH54246 build-sha: 7544c1ea combined-build-number: 998 core-build-number: 3938

The additional fields are all of the text that follow the second (“build:”) line version number. If we remove the following lines from the file – /usr/lpp/IBM/zoautil/bin/.buildinfo – and drop the “3938” after the version detail on the build line, so that it looks like this:

# Generated file. Do not edit! build: 2023/09/20 17:20:50 CUT V1.2.5.0 # build: 2023/09/20 17:20:50 CUT V1.2.5.0 3938 # git-commit: dfc0ed10 # APAR: PH54246 # build-sha: 7544c1ea # combined-build-number: 998 # core-build-number: 3938

We know it says “Do not edit!”, but we want things to work. We’ve retained the original data in commented out lines, so if IBM require the information, we just need to remember this modification!
Having made the change, zoaversion reports:

IBMUSER:/Z25A/usr/lpp/IBM/zoautil/bin: >zoaversion 2023/09/20 17:20:50 CUT V1.2.5.0

And the “zos_gather_facts” for the “ipl” subset works and – for us on our zPDT – reports:

TASK [Collecting z/OS config] ******************************************* ok: [zpdt]{

TASK [What did we get?] ******************************************* ok: [zpdt] => {
"msg": {{
"ieasym_card": "ZZ",{
"iodf_unit_addr": "0A82",{
"ipaloadxx": "ZZ",{
"load_param_device_num": "0A82",{
"load_param_dsn": "SYS1.IPLPARM",{
"master_catalog_dsn": "CATALOG.Z25D.MASTER",{
"master_catalog_volser": "D5SYS1",
"nucleus_id": "1",
"operator_prompt_flag": "",
"parmlib_dsn": "ZPDT.PARMLIB",
"parmlib_volser": "ZPDT01",
"tsoe_rel": "05",
"tsoe_ver": "4"
}
}

Conclusions

Ansible is rapidly becoming the go-to task automation solution, even on z/OS. With zIIP offloading now in Python (up to 70%) and an ever-expanding set of capabilities, you should be looking at this!

1 response to “Ansible, ZOA Utilities and ZIIPy Python”

Demetrios Dimatos says:

February 5, 2024 at 7:01 am

Hi James, this is a great introduction, particularly on the Ansible modules, I really enjoyed reading this. I wanted to share with you that the issue you encountered with `zos_gather_facts` was corrected in `ibm_zos_core` version 1.6.0 which released on June 28th 2023. You can review:
– changelog entry here: https://github.com/ansible-collections/ibm_zos_core/blob/bdf9168d35466060305f1eeecaa673382e88f543/CHANGELOG.rst#L52
– release notes: https://ibm.github.io/z_ansible_collections_doc/ibm_zos_core/docs/source/release_notes.html#id8
– pull request: https://github.com/ansible-collections/ibm_zos_core/pull/797

Regarding the symbolic link for IBM Open Enterprise SDK for Python, that is needed when using Python version 3.10 or later; the reason is because the precompiled binaries included with ZOAU were compiled with Python 3.9. If you have a C compiler on your z/OS managed node, you could have gone with the PIP installation and not needed PYTHONPATH and the symbolic link (for both these, Python must be mounted RW).

These installation instructions are part of ZOAU which can be found here:
https://www.ibm.com/docs/en/zoau/1.2.x?topic=installing-configuring-zoau

The team is working to improve this over time, in ZOAU 1.3 which won’t be supported by Ansible until version 1.10.0-beta.1 (Q1 2024), ships multiple versions of the binaries, so you won’t need that symbolic link anymore.

Reply

Share this post