Gothenburg Mainframe and Continuous Integration. Jan Marek com. CA Technologies. Session S610

Jan Marek

Jan.Marek@ca.

com

CA

Technologies

Gothenburg 2015

Agenda

 Introduce continuous integration concept  What problem we were facing

 Overview of the solution

 Examples of main components  Questions & Answers

Agile software development background

Gothenburg 2015

Product Owner

Scrum Master

What does continuous integration mean?

 It is a practice in software engineering – merging all developer working copies with a shared integration mainline several times a day

 Intended to be used in combination with automated unit tests written for test-driven development

 It is a continuous application of quality control as opposed to traditional practice of doing quality control after completing all development

 So it perfectly fits the idea of iterative software delivery

Continuous integration workflow

Benefits and costs

 Current build available for demo or actual delivery to end user

 Avoids last-minute chaos at release dates and ensures compatibility of several pieces done by different teams

 Integration bugs are detected early and are easy to track down  Enforces discipline of frequent automated testing

 Constructing an automated test suite requires a considerable amount of work  There is also some work involved to set up and maintain a build system

Problems we were facing

 Legacy code with low test coverage  ISPF as well as pure batch applications  Legacy test suite

– Complex batch job streams parameterized by PROCs

– It would take moths to understand the logic and extend the coverage  Iterative software delivery which required frequent robust testing

What we wanted

 GUI for test libraries and results

– Single interface to run, track and check test results  Ability to reuse the legacy batch jobs

 Ability to build new tests more quickly  Simple setup of the testing environment

– Various stages, releases, subsystems  Simple concept

– Not re-inventing the wheel

What we have built

 Framework written in Python (some classes in Java) that simplifies writing of scripts and automates testing of mainframe applications

 Test ISPF applications, batch programs and prepare data for tests using SQL and DB2 commands and utilities

 Main features

– Simple access to DB2 via SQL without hardcoded values – SSID, userid and so – Simple submitting of jobs from the framework and access to their outputs

– Simple access to fields in ISPF panels by their ID (based on label or ISPF panel source code)

Languages

 Python

– Designed for code readability

– Express concepts in fewer lines of code than in languages such as C++ or Java – A bit slower than Java programs

– Best suited as a “glue” languages  Java

– Intended for portability - write once and run anywhere

– Better characterized as a low-level implementation language  Jython

– Python designed to run on the Java platform

High level architecture

Gothenburg 2015

x3270 FTP JDBC driver Subversion repository Jenkins server Local computer JES

Subversion – best practicies

 Frequent commits

– Commit early and often to find issues as soon as possible  Policy for trunk

– No syntax error on trunk, framework unit tests must pass  Commit related code changes together

– Do not commit unrelated code in the same commit  Do not commit build artefacts

– Use ignore list

 Use comments when you commit

– Comments help others understand changes

 Post commit event that sends an email to your development mailing list  Use authentication

Connection to DB2 via JDBC driver

 Used to execute SQL queries and commands

 Ensure you have the following jars in your classpath – db2jcc.jar

– db2jcc_licence_cu.jar

– Part of any DB2 LUW 10.5

 JayDeBeApi allows you to connect from Python to database using JDBC

Gothenburg 2015

import jaydebeapi

connection = jaydebeapi.connect('org.hsqldb.jdbcDriver',

['jdbc:hsqldb:mem:.', 'SA', ''], '/path/to/hsqldb.jar',)

x3270

 IBM 3270 terminal emulator

 Runs on Linux, Mac OS X, Solaris, Cygwin, Windows

 Runs over a TELNET connection emulating either 3279 or 3278  Available in several different forms

– x3270 is for use on X graphics displays – wc3270 is the Windows console version – s3270 is for screen-scraping scripts – pr3287 is for printer emulation

s3270

 Implements RFCs 2355 (TN3270E), 1576 (TN3270) and 1646 (LU name selection)

 Supports IND$FILE file transfer

 Opens a telnet connection to an IBM host and allows a script to control the host login session

 Supports the telnet-passthru service to allow outbound connection through a firewall machine

py3270

 A Python interface to x3270 IBM 3270 terminal emulator  It provides an API to x3270 or s3270 subprocess

 Contains the Emulator class which we have extended

– Main benefit – automatic recognition and naming of input fields – General functions such as login, fields recognition

– TSO and ISPF commands, reading of panelid and messages – DB2 and DB2 Tools specific commands and options

py3270 – example

from py3270 import Emulator

# use x3270 so you can see what is going on em = Emulator(visible=True) # or not (uses s3270) em = Emulator() em.connect('3270host.example.com') em.fill_field(17, 23, 'mylogin', 8) em.fill_field(18, 23, 'mypass', 8) em.send_enter()

# maybe look for a status message

if not em.string_found(1, 2, 'login succesful'): abort()

# do something...

# disconnect from host and kill subprocess

FTP to JES

 Alternative to RJE

 Submit jobs and retrieve the results using TCP/IP FTP  JCL must be created on the submitting platform

– SITE JESLRECL=xxx (up to 254)

 The job name must match the user’s userid plus one character  (QUOTE) SITE FILETYPE=JES to connect to JESPLEX

 PUT to submit job, GET to retrieve the output from the spool

FTP to JES – commands example

ftp> quote site filetype=jes 200 Site command was accepted ftp> put test.jcl

200 Port request OK.

125 Sending Job to JES Internal Reader FIXrecfm 80 250-It is known to JES as JOB04661

ftp> dir

200 Port request OK. 125 List started OK

SYSLBDT JOB04661 OUTPUT 4 Spool Files

GET JOBnnnnn.s where ‘nnnnn’ is the job number from the ‘ls’ or ‘dir’ ftp subcommand and ‘s’ is the SYSOUT file sequence number.

GET JOBnnnnn.X where ‘nnnnn’ is the job number and ‘X’ is used to indicate to retrieve ALL of the JOB’s

SYSOUT into one file. The ‘X’ may be specified

FTP to JES – FTP class

 FTP class which uses the FTP to JES interface to – Submit job

– Wait for job to be completed – Find spool files

– Get spool files

– Parses JES system messages – Create/Update/Delete datasets

FTP to JES – Get jobs

 Python example

 Output from FTP

Gothenburg 2015

self.ftp.sendcmd("SITE FILETYPE=JES")

reply = self.ftp.sendcmd("SITE JESJOBNAME=%s JESSTATUS=%s JESOWNER=%s" % (USER1*, ALL, USER1)) assert reply.startswith("200 ")

ftp> dir EZA1701I >>> PORT 127,0,0,1,4,10

200 Port request OK. EZA1701I >>> LIST 125 List started OK for JESJOBNAME=USER1*, JESSTATUS=ALL and JESOWNER=USER1 EZA2284I JOBNAME JOBID OWNER STATUS CLASS EZA2284I USER1 TSU00025 USER1 OUTPUT TSU ABEND=222 3 spool files EZA2284I USER1A JOB00209 USER1 OUTPUT A ABEND=806 3 spool files EZA2284I USER1 JOB00201 USER1 OUTPUT A RC=0000 5 spool files EZA2284I USER1J JOB00208 USER1 OUTPUT A (JCL error) 3 spool files EZA2284I USER1 JOB00193 USER1 OUTPUT A RC=0000 5 spool files EZA2284I USER1A JOB00200 USER1 OUTPUT A ABEND=806 3 spool files

FTP to JES – Create dataset

 Python example

Gothenburg 2015

self.ftp.sendcmd("SITE FILETYPE=SEQ") self.ftp.sendcmd("SITE MGMTCLASS=mgmtclass") self.ftp.sendcmd("SITE RECFM=recfm") self.ftp.sendcmd("SITE PRIMARY=primary") fp = StringIO("") reply = self.ftp.storlines("STOR dsn", fp) assert reply.startswith("250") '''

Use SITE subcommand to set DCB parameters and STOR command to create dataset

Use STOR with fp to write into sequential dataset

Use SITE DIRECTORY=dirblocks with self.ftp.sendcmd("MKD dsn") to create PDS/PDSE '''

Continuous integration server

 Free, open source continuous integration tool written in Java  Server-based system running in a servlet container

 It supports CVS, Subversion, Git, RTC and so

 Builds can generate test reports in various formats

 Jenkins can display the reports and generate trends and render them in the GUI

Jenkins – projects and test suits

Jenikns – test results

Questions & Answers