American Express Data Center Migration Presentation to the TPF Users Group Washington, DC

(1)

American Express

Data Center Migration

Presentation to the TPF Users Group

Washington, DC

(2)

Agenda

• Context

• Point of Departure

• New Data Center Set-Up

• New Data Center Validation

• Migrations

(3)

Point of Departure - CAS/TPF Environment

Three Processors (2) IPC1, (1) MDC

> IPC1 Production and Dev/Test are split

> Dev/Test provides a production back-up platform

> MDC for Disaster Recovery Production and Dev/Test share Six DASD Sub Systems (3) IPC1, (3) MDC

> Production and Dev/Test are striped across all Sub Systems > West and ADC share some volumes with TPF

Four Tape ATLs (2) IPC1, (2) MDC

> Production and Dev/Test share all tape

> West and ADC share Tape ranges with TPF 100% IP network as of March 2012

Shared Console environment across LPARs

> Sharing limited by LPARs being fixed to separate processors

(4)

Point of Departure: 2012

IGS TPF Install and Testing Plan --- March

> Provide WWCAS a Detail Timeline and Migration Plan EMC/IGS detail install plan --- March

> EMC to reflect back IGS/AMX Requirements for IPC1 and MDC hardware, Enginuity microcode, all scripts, EMC Product Suite for TPF, and detail copy design

> EMC to reflect back IGS/AMX Requirements for IPC2 hardware, Enginuity microcode, EMC Product Suite for TPF and copy all scripts, and detail copy design

Replication Network Detail Planning --- March

Update WWCAS on Replication planning provided & Timeline --- April IGS/EMC IPC1/MDC Migration Methodology Test plan --- April

Channel Extension install and zoning IPC1, MDC, IPC2 --- May

EMC configuration and Product Suite for TPF changes for Test Plan --- May Verify 7.0.0.4 and Cascaded SRDF in the IPC1/MDC environment --- June

(5)

Point of Departure: CAS/TPF DASD Environment

zTPF Dupe SHRP zTPF Dupe SHRP AR2 zTPF Dupe SHRP CLN R1 IPC1 PHX MDC MSP zTPF Prime R2 WK/AZ ATE zTPF Prime AX AY AZ SHRT VM AT SHRT VM SHRT VM SRDF/A SRDF/A +

Local Box TimeFinder Copy

Local or Remote Box SRDF/A or SRDF/DM Copy TPF Records striped across 3 VMAX using SRDF/DM +

Red or Black, SRDF or TimeFinder, local or remote, suspended copy Immediate Tape file copies

(6)

EMC Migration Method Validation: June 30, 2012

zTPF Dupe SHRP zTPF Dupe SHR P AR21 Temp zTPF Dupe IPC1 PHX MDC MSP Temp zTPF Prime R2 Temp zTPF AX zTPF Prime AX AY AZ SHRT VM AT SHRT VM SHRT VM SRDF/A SRDF/A + I9, I10 M16

* Set-up DR Environment to validate IPC2 requirements * This can be made permanent for the Migration Method

zTPF Dupe SHR P CLN R1 zTPF Prime R2 WK/AZ ATE + R21

(7)

Set-up IPC2 Environment: 2012

EMC configuration input to IGS Hardware Management --- July

> IPC1/MDC/IPC2 Microcode upgrade and patches

Copy Network install IPC1, MDC, IPC2 --- July

Channel Extension install and zoning IPC1, MDC, IPC2 --- July EMC DASD and BIN File upgrades IPC1, MDC --- August

EMC delivery of all scripts --- September IBM install W9 processor & Tape --- October IGS install SuperVision Servers --- October

EMC DASD and BIN File install IPC2 --- October

> IPC1/IPC2 IOPS 300k -- Projected

> IPC2 Double bandwidth of IPC1 (RFs in VMAX)

(8)

Introduction of IPC2 Hardware: October 31, 2012

zTPF Dupe SHRP zTPF Dupe SHRP AR21 zTPF Dupe SHRP CLN R1 IPC1 PHX MDC MSP zTPF Prime R2 WK/AZ ATE zTPF Prime SHRP AX AY AZ SHRT VM AT SHRT VM SHRT VM SRDF/A SRDF/A IPC2 zTPF Prime zTPF Dupe SHRP SHRT VM 24 48 WK1 SHRT VM + W9 I9, I10 M16

(9)

TPF Systems Validation: 2012

Fully Functional W9 Environment --- October - November > First VM/TPF copy

• Cascaded SRDF/DM from MDC AR21 Dupe to IPC2 R2 Dupe • TimeFinder Clone from IPC2 R2 Dupe to IPC2 AX Clone (R1) • SRDF/DM from IPC2 AX R1 Dupe to IPC2 R2 Prime

• SRDF/DM from MDC VM/SHRT R1 to IPC2 VM/SHRT R2 • TimeFinder VM/SHRT IPC2 R2s to IPC2 VM/SHRT Clones

> First VM/TPF IPL

• Hardware Verification, Processor, DASD, Tape

TPF Testing/Validation --- November - December > TPF DB Utilities

• Test Cases

• Tape copy delay timings and copy start

> Application Utilities

• Test Cases

> Repeatable TPF Migration Copy Function

• Developed and Documented

(10)

IPC2 Validation: December 31, 2012

zTPF Dupe SHRP zTPF Dupe SHRP AR21 zTPF Dupe SHRP IPC1 PHX MDC MSP zTPF Prime WK/AZ ATE zTPF Prime AX AY AZ SHRT VM AT SHRT_VM SHRT VM SRDF/A SRDF/A IPC2 zTPF Prime zTPF Dupe SHRP SHRT VM 48 24 WK1 SHRT VM R2 + W9 I9, I10 M16 + + +

(11)

WWCAS IPC2 Validation: January – July, 2013

TPF Stand-alone Network Start --- January

> Validate Network connectivity

> Begin TPF Migration Copies per WWCAS requests

Additional Processor Install/Validation --- February Network Connectivity Validation --- February -- May

> West, East, International, CAS Transactions testing in IPC2 > Individual GAN, STRATUS, traffic generators

> Outage Timing Runs, IPC2 DR like verification exercise

Production Network Migration Testing --- June & July (after MG6 & MG7)

> Final Tape Copy Validation, remote shared tape > Final West Validation, remote shared DASD > Final TPF Migration Validation

(12)

DC2015 Risk Platforms Migration Calendar

4 11 18 25 1 8 15 22 1 8 15 22 29 5 12 19 26 3 10 17 24 31 7 14 21 28 5 12 19 26 2 9 16 23 30 6 13 20 27 MG 4.2 Midrange 4.2 MG 4.3 Midrange 4.3 MG 4.4 Midrange 4.4 MG 5A Midrange 5A MG 5B Midrange 5B MG 6A Mainframe (DIPJ) 6A 6A 6A Mainframe (PPRD/EPRD) 6B 6B Midrange 6B 6B MG 6C Midrange 6C Mainframe (West/LPRD) 7A 7A Midrange 7A 7A MG 7B Midrange 7B Mainframe (EAST/zData) 8A 8A Midrange 8A 8A MG 8B Midrange 8B MG 8C Mainframe (CAS/TPF) 8C 8C

Primary Migration Event Application Freeze- Primary Migration Date (2 wks prior,3 days post)

Contingency Migration Event Application Freeze- Contingency Migration Date

DC2015 Migration Calendar- 2013

January February March April May June July

MG 6B

MG 7A

MG 8A

August September

Mar 22 Risk Rules Engine (WinARES)

Apr 12 Credit Review Online (iRisk ), Enterprise MIS, AMC, Enterprise Model Mgr, Risk Change Mgmt & BNR portals, CDIT

Apr 26 Institutional and OPEN UW (CRUSE, iCRUSE)

ROP, AboutRisk _{Jun 7 Credit Bureau Reporting (CBR), Credit and} Collections (WCC), Global Account Monitor (GAM),

Global Acquisition (GNA), Card Number Generation (CANS), Risk Assessment (GRMS)

Aug 9 Credit Bureau Reporting (CBR), Credit and Collections (WCC), Global Account Monitor (GAM), Global Acquisitions (GNA)

Card Number Generation (CANS), Risk Assessment (GRMS)

Jul 19 IDN Warehouse, IDN Portal,

Jul 12 Credit Bureau Reporting (CBR), Credit and Collections (WCC), Global Account Monitor (GAM),

Global Acquisitions (GNA), iBureau

Aug 30 Authorizations (CAS)

Region Migration Date EST Outage Time Outage Duration Local Time

IPCW 7/13/2013 22:00 – 14:00 EST 16 hours 00:01 – 16:00 MST

IPCE 8/11/2013 00:01 – 16:00 EST 16 Hours 00:01 – 16:00 EST

EPRD 6/8/2013 & 6/9/2013 14:00 – 06:00 EST 16 Hours 19:00 – 11:00 GMT

PPRD 6/8/2013 & 6/9/2013 14:00 – 06:00 EST 16 Hours 04:00 – 20:00 UTC

Jun 16 IDN Analytics, Big Data

Aug 16 Sphere (Global Decision Engine, Global Risk Fulfillment

(13)

Mainframe Migration Downtime (MST-PHX Times)

Soft Launch (DIPJ) International (EPRD, PPRD) zLinux WEST zLinux EAST zLinux CAS/TPF Move Group 6A May 11-12 Move Group 6B June 8-9 Move Group 7A July 13-14 Move Group 8A Aug 10-11 Move Group 8C Aug 31

START START TIME

(MST-PHX) END

END TIME* (MST-PHX)

Saturday – May 11 18:00 Sunday - May 12 4:00

Saturday – June 8 12:00 Sunday – June 9 2:00

Sunday– July 14 00:01

(midnight) Sunday – July 14 16:00

Saturday– August 10 21:00 Sunday – August 11 13:00

START START TIME

(MST-PHX) END END TIME (MST-PHX) Saturday – August 31 2:00 Saturday – August 31 6:00

Saturday – July 13 23:00 Sunday – July 14 12:00

LPRD

UROC START START TIME

(MST-PHX) END

Saturday August 10 00:01

(midnight) Saturday August 10 13:00

(14)

CAS/TPF Migrations: June – August, 2013

CAS/TPF Migration Dry Run Testing -- June -- July West Migration to IPC2 -- July 12, 2013

> Remote Shared DASD Automation

CAS/TPF Migration to IPC2 -- August 31, 2013

> Run EMC Product Suite for zTPF in IPC1 until commit to IPC2 > Back-up the EMC Control Tables

> Sync VM in IPC2 via SRDF/A from IPC1 and IPL VM in IPC2 > Activate SRDF/DM copy from MDC to IPC2

> Ensure all copies are very close to in-sync • Cycle TPFA in IPC1 to 1052 state

• Start a new cycle and stop copies

> When all copies are 100% in-sync • Perform Pend/Drop on TPFA SRDF/A • Suspend

• SWAPair • Split • Suspend • Delpair

(15)

Summary of Credit Authorizations (CAS) Mainframe Migration

Pre-Migration Migration Post

Migration

Infrastructure Hardware and data replication enhancements

have been implemented

Integration Developed integrated migration plan working with

all vendor partners Change Restrictions All changes will be reviewed

weekly with development teams for freeze impacts

DATA Integrity Process in place to ensure

all updates are applied to the new hardware

Outage Time Scheduled 60 minutes starting on Saturday @ 02:00MST Repeatable Developed a repeatable testing process to validate

data, software, and hardware in the new

datacenter

Impacts

Servicing, ATM, PIN@POS functions will be unavailable during the migration change

window

Validate Data

All teams will be engaged to validate data integrity

Uplift

Upgraded disk arrays used for authorizations to avoid additional cost and outages Validate Apps

All teams will be engaged to validate capabilities

Stand-In Distributed platforms provide authorizations during planned outage

Fallback

Processes and scripts ready to execute in case of fallback decision Validate Connectivity

All teams will be engaged to validate network and intersystem connectivity Authorizations Move Group 8C Aug 31 START START TIME (MST-PHX) END Outage Duration

(16)

Move Group 8C Move Day Timeline – TPF/CAS on August 31

00:01AM CHECK POINT – All Pre-Work Confirmed

4:30AM BSD VP Sign-off Migration Complete

Saturday, August 31

6:00AM 10:00AM CHECK POINT Monitor/TPF health 1:00AM Open Migration Bridges 1:30AM Quiesce batch and

Drain copy data to IPC-2 1:50AM CHECK POINT Confirm all is ready

for shut down 2:00AM Shut down TPF 2:05AM Drain and Stop SRDF/A 2:15AM Complete data synchronization 2:25AM Split IPC-2 data from the copy process 2:40AM CHECK POINT Confirm ready to start TPF 3:00AM Start TPF in IPC-2 3:15AM CHECK POINT Confirm TPF is up and available 3:00AM Systems, BSD, and Business verification of migrated data and

network connectivity

Additional Check Points:

9/1 – 3:00am, 8:00pm 9/2 – 8:00am, 8:00pm

5:00AM Migration GO/NO GO Checkpoint decision complete

6:00AM Migration Complete

(17)

August 2013 TPF Performance – IPC1

•

Mills per Msg: 4.16

•

MIPS per Msg: 2.16

•

ECB’s in Use: 173.15

•

VFA Efficiency: 76.7%

•

DASD Service Time: 0.973ms

(18)

TPF Migration: August 31, 2013

zTPF Dupe SHRP zTPF Dupe SHRP AR21 zTPF Dupe SHRP IPC1 PHX MDC MSP zTPF Prime SHRP WK/AZ ATE zTPF Prime AX AY AZ SHRT VM AT SHRT VM SHRT VM SRDF/A SRDF/A IPC2 zTPF Prime zTPF Dupe SHRP SHRT VM 48 ₂₄ WK1 SHRT VM R2 W9, W10 I9, I10 M16 + + + +

(19)

CAS/TPF Migration Success: August 31, 2013

•

To minimize downtime and customer impact, the CAS

and IBM teams completed a total of four dry runs in

preparation for the migration.

• Automation and scripting were used extensively and the

duration of the actual CAS outage was minimized to only

27 minutes (2:02am-2:29am MST).

• As planned, GAN stood-in for CAS during that time

period and Payment Awareness provided stand-in logic to

ensure payments were not interrupted.

• All technical, application and business validation

activities were completed at 3:51am MST with no

significant problems identified.

• The CAS/TPF Mainframe environment consisted of:

• 46 applications

• 1 LPAR

• 6,300 MIPS

(20)

CAS/TPF Migration Commit: August 31, 2013

CAS/TPF Migration -- August 31

Stop running EMC Product Suite for zTPF in IPC1 > Run EMC Product Suite for zTPF in IPC2

> Set-up CAS/TPF DR copies

• Create IPC2 to MDC SRDF/A Groups from IPC2 Dupes (no clear)

• Create MDC SRDF Groups from Dupes to Primes • Create MDC TimeFinder Clone Sessions

• Create IPC2 TimeFinder Clone Sessions • Create IPC1 SRDF Sessions

> Commit Decision for CAS/TPF to remain in IPC2 > Clean-up & build new CAS/TPF DR copies

• Clean-up any SRDF/TimeFinder session not needed • Delpair SRDF TPFA

• Terminate IPC1 TimeFinder Clone Sessions • Remove IPC1 TimeFinder and SRDF Groups > Initiate CAS/TPF DR copies from IPC2 to MDC

(21)

TPF Migration Commit: August 31, 2013

AR21 zTPF Dupe SHRP IPC1 PHX _{MDC MSP} zTPF Prime WK/AZ ATE AT SHRT VM AY SHRT VM SHRT VM SRDF/A IPC2 zTPF Prime zTPF Dupe SHRP SHRT VM 48 24 WK1 SHRT VM R1 zTPF Dupe SHRP AX AZ SRDF/A zTPF Dupe SHRP zTPF Prime R2 + W9,W10 I9, I10 M16 +

(22)

CAS/TPF Migration Success: August 31, 2013

CAS/TPF Migration completed and BAU

---August 31, 2013

>Normal operation in IPC2 / Normal DR support in

MDC

• TPF Production back-up copies created in

IPC2

• TPF DR copies in MDC created from IPC2

• TPF Test Systems copies in IPC1 created from

IPC2

• VM/data copies in IPC2 from IPC1 to support

Production TPF issues only

(23)

TPF Migration Success: August 31, 2013

zTPF Dupe SHRP IPC1 PHX _{MDC MSP} zTPF Prime WK/AZ ATE AT SHRT VM AY SHRT VM SHRT VM SRDF/A IPC2 zTPF Prime zTPF Dupe SHRP SHRT VM 48 24 WK1 SHRT VM R1 zTPF Dupe SHRP AX AZ SRDF/A zTPF Dupe SHRP zTPF Prime + W9,W10 I9, I10 M16 + + +

(24)

September 2013 TPF Performance – IPC2

•

Mills per Msg: 3.04 (-26.9%)

•

MIPS per Msg: 2.32 (+6.8%)

•

ECB’s in Use: 92.72 (-46.5%)

•

VFA Efficiency: 76.5% (- 0.2%)

•

DASD Service Time: 0.467ms (-51%)

(25)

March 2014 TPF Performance

CAS Authorizations TPF Time

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 Series1

(26)