DevOps for Mainframe for IBM Pulse Conference

DevOps for the Mainframe
Leveraging Continuous Integration, the cloud, and beyond to
deliver z/OS applications

David Myers, Product Manager, Continuous Delivery for
Mainframe and Mobile Software

Rosalind Radcliffe, Distinguished Engineer, Enterprise
Modernization Solution Architect

© 2012 IBM Corporation

Please note

IBM’s
statements
regarding
its
plans,
direc3ons,
and
intent
are
subject
to
change
or
withdrawal

without
no3ce
at
IBM’s
sole
discre3on.

Informa3on
regarding
poten3al
future
products
is
intended
to
outline
our
general
product
direc3on

and
it
should
not
be
relied
on
in
making
a
purchasing
decision.

The
informa3on
men3oned
regarding
poten3al
future
products
is
not
a
commitment,
promise,
or

legal
obliga3on
to
deliver
any
material,
code
or
func3onality.
Informa3on
about
poten3al
future

products
may
not
be
incorporated
into
any
contract.
The
development,
release,
and
3ming
of
any

future
features
or
func3onality
described
for
our
products
remains
at
our
sole
discre3on.

Performance
is
based
on
measurements
and
projec3ons
using
standard
IBM
benchmarks
in
a

controlled
environment.

The
actual
throughput
or
performance
that
any
user
will
experience
will

vary
depending
upon
many
factors,
including
considera3ons
such
as
the
amount
of

mul3programming
in
the
user’s
job
stream,
the
I/O
conﬁgura3on,
the
storage
conﬁgura3on,
and
the

workload
processed.

Therefore,
no
assurance
can
be
given
that
an
individual
user
will
achieve
results

similar
to
those
stated
here.

2

New era systems integrate existing operational systems
with rapid delivery of new client-facing apps

Social, Local, Mobile
Smart Infrastructure Analytics

CRM
HR

Legacy

ERP

DB

Manage workloads Rapid innovation User experience and
and maintain security in the cloud mobile management

3

3

Evolving customer and market expectations

Capabilities and User Experience Today Emerging

Systems of Record Systems of Engagement (+ Record)
Primary Workload Types Transactional Big Data, Analytics, Mobile/Social Channels

Time to Value Planned Opportunistic

Delivery Model Planned Incremental (DevOps)

Development and Operations Team Sizes 100s and Costly 10s with built-in DevOps automation

Release Frequency Months to Years Hours to Days, based on business opportunity

Integration Frequency Weeks Continuous

Service Sourcing Develop Consume and Assemble (Public and Private)

Operational Model Systems Management Built in to application, Recovery Oriented Computing, Continuous Availability

Infrastructure Deployment Days Minutes
4
Risk Profile Big-Bang (High Risk) Incremental

DevOps is…
A set of principles and values that facilitate
collaboration across disciplines to…

1. Enable rapid evolution of deployed business services
2. Reduce risk, decrease cost, and improve quality across
the portfolio
People

Process

DevOps Principles
§  Collaborate across disciplines Tools

§  Develop and test against a production-like system
§  Deploy frequently using repeatable and reliable
processes
§  Continuously monitor and validate operational quality
characteristics
§  Amplify feedback loops
5

Solution: A Continuous Delivery Pipeline

Development Testing Staging Production
Business
Owners Customers

Ensure applications are production-ready throughout the lifecycle
and can be released at any time while minimizing rollback due to quality issues

•  Validate
on
more
produc3on-‐like
•  Standardiza3on
on
processes
and
assets

condi3ons
earlier
between
Dev
and
Ops

•  Automate
hand-‐oﬀs/promo3ons
to
•  Automated
monitoring
and
dashboarding

increase
velocity
through
the
diﬀerent
of
quality
and
performance
against
service

stages
level
agreements
at
mul3ple
stages

6

Mainframe Delivery Pains…

§  Multiple teams working across §  Too much bad code going into
restricted dev and test capacity test and production causes crit
lead to conflict, delays, or bad sits and emergency fixes
test results in shared
environments

§  Bottlenecks due to inefficient
§  Complex and manual
communications between
management and configuration disparate platforms and teams
tasks result in errors and delays (Dev/Test - System Programmers;
mobile – distributed-mainframe)

7

…solutions from IBM

Play
Defense

§ Provide
cheap,
isolated,
development
§ Enforce
base
quality
standards

and
test
environments
for
project
automa3cally
prior
to
promo3on

teams

–  Ra3onal
Development
and
Test
Environment
–  Ra3onal
Test
Workbench

–  Ra3onal
Test
Virtualiza3on
Server
–  Ra3onal
Quality
Manager

–  SmartCloud
Provisioning

–  SmartCloud
Applica3on
Monitoring

–  Cloud
Ready
for
Linux
on
System
z
–  Omegamon

§ Automate
consistent
build,
conﬁgure,
§ Improve
communica3on
and

and
deploy
processes
across
all
stages

collabora3on
with
cross-‐plaVorm

–  Ra3onal
Team
Concert
release
planning

–  SmartCloud
Con3nuous
Delivery
–  IBM
Collabora3ve
Lifecycle
Management

–  SmartCloud
Orchestrator
–  Smart
Cloud
Control
Desk

Go
on
Oﬀense

8

DevOps for the common mainframe delivery cycle

1

2
2

Dev
Test

Dev
Test

Dev
Test
Staging
Prod

4

3

Development" Test" System
Programmer"

9

Typical z/OS Testing Architecture
Organized by project team, vertically scaled, sharing resources, limited automation

z/OS
Problems Encountered
Test LPAR
Project
Team 1. Shared resources combined
[April Maintain] with overlapping schedules can
App elicit conflicts, impede
innovation and slow code
Project delivery
Team
[Prototype SOA]
2. Coordination of environmental
Test
changes and releases cause
App bottlenecks, delays and
Project Data

Team additional overhead
[June New Func]
3. Shared test data is difficult to
…

manage and can lead to over
testing or incorrect test results
Project
Team App
[Dec Sys Upgrade]

10

10 10

Rational Development and Test Environment for
System z
The ultimate in modern application development for System z

COBOL,
PL/I,
C++,
Java,
EGL,
Batch,

Assembler,
Debug
Tool

IMS

DB2
CICS
WAS MQ
RDz user
RDz user z/OS

x86
PC
running
Linux

RDz user
ISPF user

RDz user
RDz & ISPF user

§  Liberate
developers
to
rapidly
prototype
new
applica3ons

§  Develop
and
test
System
z
applica3ons
anywhere,
any3me!

§  Free
up
mainframe
development
MIPS
for
produc3on
capacity

§  Eliminate
costly
delays
by
reducing
dependencies
on
opera3ons
staﬀ

Note:
This
Program
is
licensed
only
for
development
and
test
of
applica3ons
that
run
on
IBM
z/OS.
The
Program
may
not
be
used
to
run
produc3on
workloads
of
any
kind,
nor
more
robust

development
workloads
including
without
limita3on
produc3on
module
builds,
pre-‐produc3on
tes3ng,
stress
tes3ng,
or
performance
tes3ng.

11

Decouple subsystem teams using Green Hat
Improve
tes+ng
with
Green
Hat
service
simula+on

Simulate subsystem dependencies
–  Test impact of latency
–  Test application response to unresponsive
services

Application changes
Create a stable test environment being tested
–  Simulate “public” services
–  Reduce capacity requirements on
infrastructure
–  Improve security by reducing access from
external services
Databases Mainframe Third-party
applications Services
virtualized services
Ra3onal
Test
Virtualiza3on
Server

12

Testing with dependency virtualization

Controlled large system testing by isolating components under test
•  Easier problem determination
•  Lower test environment capacity requirements
•  Improved component quality
Phase 1 Phase 2 Phase 3
Test
Case

Test
Case

Test
Case

IMS
Data 3rd Party
IMS Data 3rd Party
Access Call IMS Data 3rd Party
App
Access
App

Call
Access Call
App

CICS
CICS
Commarea CICS
Commarea
Call Commarea
Call
Call

Virtual
Virtual

Virtual
Services

Services
Services

13

Provision new dev / test systems in minutes
•  Fast
VM
provisioning
for
near-‐instant

deployment
of
100s
of
virtual
machines
and

scale
based
on
business
needs

•  Rich
set
of
web
interfaces
into
the
cloud
that

can
be
driven
by
a
user
or
scripts
for
complete

automa3on

•  Reliable,
non-‐stop
cloud
capable
of

automa3cally
tolera3ng
and
recovering
from

soeware
and
hardware
failures

•  Near-‐zero
down=me
due
to
faults,
hypervisor/
management
soeware
upgrades
or
addi3on/
removal
of
hardware

Func3on
Test
Performance

Test

Requested
VMs
will
be
up

and
running
under
a

minute
using
standard
HW

14

Fail fast and drive better quality downstream
Reduced delivery time, end-to-end visibility of test activities, safer and faster upgrades (V2V)

Automated, Reusable
continuous testing integration
with isolated Continuous Integration for z tests simplify
development and testing across
test environment layers
Check
In
+
Build
+
Deploy
+
Run

Code
Automated
Tests

Change

Promote

Request
To
QA

Frequent, rapid
feedback earlier Developer Higher quality
to improve

going into QA
quality Rapid
Feedback

§  Fast, dependable, automatic feedback speeds time to market
§  Lower cost of application testing using off-mainframe z/OS test environment
§  Enables confidence by automatically tracking and promoting code health

15

Continuous Integration for System z Scenario

System z
Test
ests>
pp+zUnit T Environment
<Deploys A (RD&T or LPAR)

<Drives>
SCM/CI <Starts Tests> Test
Server Execution
(Rational Team Manager
Concert) (Ra3onal
Quality

<Results> Manager)
Test
Automation
(Rational Test
1.  Check-in code 5.  Kick-off Automated Workbench)
2.  Build code and zUnit Test Plan
6.  Run automated
tests interface tests against
3.  Deploy build results 8.  Report test results Test Environment
and test data to Test in dashboard/build 7.  Mark execution
Environment results/defect records Pass/Fail in
4.  Execute zUnit Tests records in CI server. Test Execution 16

Manager
16

Separate tools makes collaboration inefficient or
difficult...

§ Each tool came with its own Tool A
Dev work Items Tool B
•  UI - Web and desktop
presentations of views and DB
Release
Planning
UI
tasks
•  Logic – Workflow, process, LOGIC
DB
UI
search, query, scale, security Tool E
and collaboration LOGIC

•  Storage – Availability,
SCM
traceability DB
UI

•  Privacy, backup/archive
LOGIC
§ Resulting in... DB
UI
•  Brittle integrations
LOGIC
•  Silos everywhere DB
UI Tool C
•  High cost to maintain and
Tool D LOGIC Reporting
administer
•  Proprietary API's Incident
Management

17

Tie development release to operational release
Ra3onal
Team
Concert
Release
Plan
and
work
items

SmartCloud
Control
Desk

18

Delivery Organized for Flexibility and Speed
Organized by application team, horizontally sliced, dedicated resources, highly automated

z/OS LPAR
Problems Encountered
Project 1.  Shared resources
Team
[April Maintain] LPAR combined with
overlapping schedules
Data
can elicit conflicts,

QA/Integra3on
Test
LPAR

impede innovation and
Project
Team slow code delivery
[Prototype SOA] RD&T 2.  Coordination of
Data
Integr.
environmental changes
Test
and releases cause
Project Data
bottlenecks, delays and
Team
[June New Func] RD&T
additional overhead
3.  Shared test data is
…

Data
difficult to manage and
can lead to over testing
Project or incorrect test results
Team
[Dec Sys Upgrade] LPAR 4.  Provisioning, managing,
and synchronizing project
Data

test environments
including data 19

19 19

Deploy application and config changes to test environments in minutes

Opera+ons
team
Development
team

establishes
deployment
makes
applica+on

pa;erns
for
use
by
IBM code
and

development
and
test
SmartCloud
conﬁgura+on

Continuous
teams
Delivery
changes

Deliver
Environments 3x
Test against
Faster!!
production-like
environments with
Applications &
middleware
AUTOMATICALLY
installed and properly IBM
configured IBM SmartCloud Workload
20
Provisioning Deployer

What are we working on in
the lab…

21

…solutions from IBM

Play
Defense

§ Provide
cheap,
isolated,
development
§ Enforce
base
quality
standards

and
test
environments
for
project
automa3cally
prior
to
promo3on

teams
–  Ra3onal
Test
Workbench

–  Ra3onal
Development
and
Test
Environment
–  Ra3onal
Quality
Manager

–  Ra3onal
Test
Virtualiza3on
Server
–  SmartCloud
Applica3on
Monitoring

–  SmartCloud
Provisioning

–  Omegamon

–  Cloud
Ready
for
Linux
on
System
z

§ Automate
consistent
build,
provision,
§ Improve
communica3on
and

conﬁgure,
and
deploy
pof
zLinux
across
collabora3on
with
cross-‐plaVorm

Cloud-‐style
deploy
rocesses

all
stages

dev
and
test
paMerns

and
z/OS
release
planning

on
RD&T
or
zHardware

–  Ra3onal
Team
Concert
–  IBM
Collabora3ve
Lifecycle
Management

–  SmartCloud
Con3nuous
Delivery
–  Smart
Cloud
Control
Desk

–  SmartCloud
Orchestrator*

Go
on
Oﬀense

22

Provisioning using Cloud

#!/usr/bin/env ruby

class DevopsDeployer
def initialize(build_url, build_id)
@log = Logger.new(LOG_FILE)
@log.level = LOG_LEVEL

@iaas_gateway = IaasGateway.new(HsltProvider.new(),
LOG_FILE, LOG_LEVEL)
@server_instance = nil
.cbl
.cpy

rtc_build_system_provider = RtcBuildSystemProvider.new(
RTC_REPOSITORY_URL, RTC_USER_ID, RTC_PASSWORD_FILE)
@build = rtc_build_system_provider.resolve_build(
build_url, ENV['buildResultUUID'], build_id)
Infrastructure

as
Code

@build_system_gateway = BuildSystemGateway.new(
.asm

rtc_build_system_provider, LOG_FILE, LOG_LEVEL)
end

def add_build_stamp
template_file = WEB_APP_ROOT +
"/app/templates/pages/page.html"
@log.info "Adding build ID stamp #{@build.id} to
.rexx

#{template_file}" CLIST

# Read in the file's contents as a string, replace
# the build_id, then overwrite the original contents
# of the file

text = File.read(template_file)
new_text = text.gsub(/{{ build_id }}/,
Source
Ar3facts

"<a href="#{@build.uri}">#{@build.id}</a>")
File.open(template_file, "w") { |file|

}
end
Source
Control

file.puts new_text

# ... Management

23

Delivery Pipeline Fundamentals

Build,
Package,

&
Unit
Test

Applica+on

.cbl
.cpy
Binaries
&

PlaGorm

Conﬁgura+on
Deploy

.asm

.rexx
CLIST

Environment

Deployable
Ar3facts
Running
System

Source
Ar3facts

Source
Control

Library

Management

24

Consolidate, standardize, simplify deployment

Enterprise
Applica3ons
have
thousands
of
disparate
parts

•  Currently
maintained
in
separate
systems,
if
maintained
at
all

•  Limited
linkage
between
systems
for
applica3on
and
conﬁgura3on
dependencies

•  Missing
assets
and
informa3on
is
rampant

DevOps
force
linkages,
automa3on,
and
standardized
packaging…

Copybooks
HTML

COBOL
JCL
Build,
Package,

WSDL
&
Unit
Test

MQ
Applica+on
Deploy

Java

Queue
Binaries
&

Def
PlaGorm

Conﬁgura+on

DB2

IMS
DBD/PSB

DDL
DB2
Plans
Environment

Deployable
Ar3facts
Running
System

Procs

Library

Source
Management
25

Create, secure, and deploy test data
Automated Build &
Deploy Test Data

-Compare z/OS

-Subset -Refresh UT

RD&T

2TB 1 GB
Test Data
z/OS

Production or
UT

LPAR

Production Clone
1 GB
Masked Test Data
Automated
Testing

Create “right-size” De-identify sensitive information Automated test data
production-like environments De-identify sensitive information
with realistic but fictional data for deployment for each
for application testing with realistic purposes
testing & development but fictional data for
testing & development purposes build and test

Optim RTC 26

Standardize z/OS region configurations to ease
deployment

Conﬁgura3on
Applica3on
•  Standard topologies exist today
(production LPARs)

•  Standardized regions which can be
Data

repeatedly deployed are rare

•  Standardized/Automated deployment of
COMPLETE system is spotty
CICS
5.1
IMS
11.1

Standard
Middleware
Palern
To adopt the cloud for testing…
Standardize and automate provisioning of
Z/OS
1.12
DB2
10.1
everything …

27

Standardize z/OS region configurations to ease
deployment

RTC/CIz Available
Application

Capabilities
Conﬁgura3on
Applica3on

load modules

Data

JCL and REXX

Test Data

CICS
5.1
IMS
11.1

Pattern/Platform
Standard
Middleware
Palern

Z/OS
1.12
DB2
10.1

28

Evolving to deploying regions … cloud-style

“Cloud”
CICS+DB2
RD&T
CICS+
IMS
Or
DB
LPAR
Region/Middleware

RD&T
Palerns
Or
Batch
LPAR

RD&T
Or
LPAR

RD&T
Or
LPAR
SmartCloud
Orchestrator

+

<SCD
for
Mainframe

Project>

29

Acknowledgements and Disclaimers:

Availability.

References
in
this
presenta3on
to
IBM
products,
programs,
or
services
do
not
imply
that
they
will
be
available
in
all
countries
in

which
IBM
operates.

The
workshops,
sessions
and
materials
have
been
prepared
by
IBM
or
the
session
speakers
and
reflect
their
own
views.

They
are
provided
for

informa3onal
purposes
only,
and
are
neither
intended
to,
nor
shall
have
the
effect
of
being,
legal
or
other
guidance
or
advice
to
any
par3cipant.

While
efforts
were
made
to
verify
the
completeness
and
accuracy
of
the
informa3on
contained
in
this
presenta3on,
it
is
provided
AS-‐IS
without

warranty
of
any
kind,
express
or
implied.
IBM
shall
not
be
responsible
for
any
damages
arising
out
of
the
use
of,
or
otherwise
related
to,
this

presenta3on
or
any
other
materials.
Nothing
contained
in
this
presenta3on
is
intended
to,
nor
shall
have
the
effect
of,
crea3ng
any
warran3es
or

representa3ons
from
IBM
or
its
suppliers
or
licensors,
or
altering
the
terms
and
condi3ons
of
the
applicable
license
agreement
governing
the
use

of
IBM
soeware.

All
customer
examples
described
are
presented
as
illustra3ons
of
how
those
customers
have
used
IBM
products
and
the
results
they
may
have

achieved.

Actual
environmental
costs
and
performance
characteris3cs
may
vary
by
customer.

Nothing
contained
in
these
materials
is
intended

to,
nor
shall
have
the
effect
of,
sta3ng
or
implying
that
any
ac3vi3es
undertaken
by
you
will
result
in
any
specific
sales,
revenue
growth
or
other

results.

©
Copyright
IBM
Corpora6on
2013.
All
rights
reserved.

§  U.S.
Government
Users
Restricted
Rights
-‐
Use,
duplica6on
or
disclosure
restricted
by
GSA
ADP
Schedule
Contract
with
IBM
Corp.

§  Please
update
paragraph
below
for
the
par+cular
product
or
family
brand
trademarks
you
men+on
such
as
WebSphere,
DB2,
Maximo,

Clearcase,
Lotus,
etc

IBM,
the
IBM
logo,
ibm.com,
[IBM
Brand,
if
trademarked],
and
[IBM
Product,
if
trademarked]
are
trademarks
or
registered
trademarks
of

Interna3onal
Business
Machines
Corpora3on
in
the
United
States,
other
countries,
or
both.
If
these
and
other
IBM
trademarked
terms
are

marked
on
their
first
occurrence
in
this
informa3on
with
a
trademark
symbol
(®
or
™),
these
symbols
indicate
U.S.
registered
or
common
law

trademarks
owned
by
IBM
at
the
3me
this
informa3on
was
published.
Such
trademarks
may
also
be
registered
or
common
law
trademarks
in

other
countries.
A
current
list
of
IBM
trademarks
is
available
on
the
Web
at
“Copyright
and
trademark
informa3on”
at

www.ibm.com/legal/copytrade.shtml

If
you
have
men3oned
trademarks
that
are
not
from
IBM,

please
update
and
add
the
following
lines:

[Insert
any
special
3rd
party
trademark
names/alribu3ons
here]

Other
company,
product,
or
service
names
may
be
trademarks
or
service
marks
of
others.

30

www.ibm.com/software/rational
© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind,
express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have
the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM
software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities
referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature
availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines
Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

31

DevOps for Mainframe for IBM Pulse Conference

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to DevOps for Mainframe for IBM Pulse Conference (20)

DevOps for Mainframe for IBM Pulse Conference