Enterprise Server Performance Analysis & Capacity Planning in High Volume Batch and On- Line Environments

(1)

Enterprise Server Performance

Analysis & Capacity Planning in High

Volume Batch and On- Line

Environments

UNITE Technology Conference Reno Nevada

22-24 September 2003

Session MCP4064 Guy Bonney

(2)

MGS, Inc.

! Software Engineering, Product Development &

Professional Services firm founded in 1986

! We provide products and services to solve business

problems:

" Software Engineering Services

" Professional Services

# Management Support Services # Consulting and Technical Services # Application Development Services # Training Services

(3)

MGS Areas of Expertise

! Principals average over 25 years industry

experience

! Junior staff - minimum 5 years experience

! 60% of experience is Unisys ClearPath NX/A

Series

! Remainder is PC, Windows™, UNIX, C,

(4)

MGS Software Products

! File Manager for ClearPath MCP/A

Series (FMA)

! SightLine™ Performance Analyzer

! SightLine Capacity Manager

! SightLine Workload Analyzer

! HVFAX High Volume Facsimile Delivery

System

(5)

(6)

IT Performance/Capacity Issues

! Business Issues

! Information Technology Management Issues

(7)

Business Issues

! Change

" Growth

$ Evolutionary

• New clients, new products, higher volumes

$ Revolutionary

• Buy a new division / product line • Merger

• Business exchange / partner collaboration • On Line customer service

• On Line payment processing or store front

" Decline

$ Lose business (cost reduction) $ Change business focus / direction $ Get bought or merged

(8)

Business Growth

! Growth in most organizations requires

Information Technology services to change

! Growth is not possible without reliable IT

service delivery and good performance

! Even without business growth IT must

(9)

IT Management Issues

! Where is the business going?

! What technologies are now available?

! What new & improved functionality do we

need to support the business?

! What service levels are needed for effective

business function?

! What capacity is required to deliver the

services needed?

! What budget is available to procure

(10)

Technical Issues

! What resources are available for service

delivery?

! How do we define the business workloads?

! What service levels are we delivering?

! How do we measure services and service

levels?

! How should we report on performance and

capacity issues?

! How do we do performance analysis?

! How do we do capacity planning?

(11)

What is Performance?

! Meeting Expectations.

! Complete Processing a Unit of Work in the

Expected Time.

" On-line transactions - Response Time

" Batch runs - Elapsed time/Deadlines

" “On-Line Batch” - Elapsed (Response) time

(12)

Monitoring the Critical Path

Message Time Message Time “Service” “Service” Measures Measures

Network Service &

Resource Measures Resource Measures Server & Server & Application Application Measures Measures

(13)

What is a Workload?

! Aggregate set of units of work to be

processed

! Everything that must be accomplished to

(14)

Workload Example

4 Jul 1999 - 3 Sep 2000

COMS Tran/Sec for All

Jul4 1999 Sep12 1999 Nov21 1999 Jan30 2000 Apr9 2000 Jun18 2000 Aug27 2000

0 20 40 60 80 100 120

(15)

What is Capacity?

! Simply the capability to process defined

amounts of work in a specified period of time

! Measured in terms of units of work

“processed”: MIPS, RPM, TPC-C transactions, MFLOPS, etc.

(16)

Performance vs. Capacity

! Capacity is the ability to process work

! Performance is the measure of workload

processing effectiveness

! Performance and Capacity must be

(17)

Performance Analysis Objectives

! Recognize performance anomalies (response

time or service time issues)

! Detect bottlenecks (measure queuing and

busy time)

! Determine cause of inadequate performance

(18)

On-Line Response Time

Time Series Analysis

22 Aug 2000 14:00 - 15:00 A19: COMS Response Avg

PGU0020 PGU0034B PGU0034 PGU0015

PGU0043B PGU0043 PGU0060 PGU0060B

14:00 14:05 14:10 14:15 14:20 14:25 14:30 14:35 14:40 14:45 14:50 14:55 15:00 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

(19)

Batch Cycle Analysis

TaskName Mix# CPUSecs IOSecs Memory IOCount PrintLns Elapsed StartTime StopTime (SFB)SFB/REPTAL0002 2004 1.6 1.3 9949 60 8 7.9 4/30/97 3:44 4/30/97 3:44 (SFB)SFB/REPACL0024 2018 111.3 93.5 15531 162324 4683 244.5 4/30/97 3:44 4/30/97 3:48 (SFB)SFB/REPACA0704 2152 56.1 25.6 16515 92726 85 71.2 4/30/97 3:52 4/30/97 3:53 (SFB)SFB/REPACA0706 2190 32.6 24.6 18091 135537 4835 52.6 4/30/97 3:53 4/30/97 3:54 (SFB)SFB/REPACA0760 2213 739.7 1916.3 19021 177340 156 2298.2 4/30/97 3:54 4/30/97 4:33 (SFB)SFB/REPAHA0760 2898 5.9 17.1 17566 1546 90 30 4/30/97 4:33 4/30/97 4:34 (SFB)SFB/REPEEA0002 2935 630.5 1274.7 21858 217908 744 2295.8 4/30/97 4:34 4/30/97 5:12 (SFB)SFB/REPAHL0033 3632 4.2 11.5 16134 3535 1120 26.2 4/30/97 5:13 4/30/97 5:13 (SFB)SFB/REPAHL0010 3648 26.8 92.3 14795 12989 1211 188.8 4/30/97 5:14 4/30/97 5:17 (SFB)SFB/REPCHA0612 3752 250 477.5 21567 53228 2057 921.4 4/30/97 5:19 4/30/97 5:35 (SFB)SFB/REPCCA0612 4032 1245 1786.9 22968 206670 9313 2696.4 4/30/97 5:35 4/30/97 6:20

TOTALS 3103.7 Secs. 8833 Secs.

(20)

Deadline Graphic Analysis

BATCH CYCLE START AND STOP TIMES

0:00 2:00 4:00 6:00 8:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00 0:00

30-Jun 10-Jul 20-Jul 30-Jul

(21)

Batch Cycle Trend Analysis

E L A P S E D TIM E F O R B A TC H C YC L E S 0 :0 0 1 :0 0 2 :0 0 3 :0 0 4 :0 0 5 :0 0 6 :0 0 7 :0 0 8 :0 0 9 :0 0 1 0 :0 0 1 1 :0 0 1 2 :0 0 1 3 :0 0 1 4 :0 0 1 5 :0 0 1 6 :0 0 1 7 :0 0 1 8 :0 0 1 9 :0 0 2 0 :0 0 2 1 :0 0 2 2 :0 0 2 3 :0 00 :0 0 7 /1 /9 7 7 /6 /9 7 7 /1 1 /9 7 7 /1 6 /9 7 7 /2 1 /9 7 7 /2 6 /9 7 7 /3 1 /9 7 E la p s e d T im e

(22)

Program Run Exception Analysis

Report Date: 8/11/97 MC0181 Exception Report for July 1997 Processing (30% > Avg. Elapsed Time) Page 1

Name Task Number CPU Secs IO Secs ReadyQ Secs Elapsed Secs Disk IO Cnt Tape IO Cnt Print Lines BOJ Date BOJ Time EOJ Date EOJ Time

MC0181 9461 5,058 902 1,152 13,893 1,065,431 4,113,762 13 7/2/97 0:56:44 7/2/97 4:48:17

MC0181 6502 5,287 854 832 16,002 987,808 4,092,475 13 7/3/97 0:36:29 7/3/97 5:03:11

MC0181 2565 5,010 1,215 184 15,796 963,257 4,081,849 13 7/5/97 22:07:06 7/6/97 2:30:21

MC0181 4232 5,367 115 278 13,571 997,660 8 13 7/29/97 22:54:21 7/30/97 2:40:33

(23)

Analyze Components

! Review utilization, queuing and throughput

rates for key components:

" CPU

" Memory

(24)

CPU Analysis

26 Jun 2000 00:00 - 3 Jul 00:00

CPU Total Pct

Jun26 Jun27 Jun28 Jun29 Jun30 Jul1 Jul2 Jul3

0 10 20 30 40 50 60 70 80 90 100

(25)

Workload CPU Distribution

21 Aug 2000 20:00 - 22 Aug 20:00

MCP Critical Path Nightly Batch

Online Processing Adhoc Batch Processing

Aug21 20:00 Aug22 00:00 Aug22 04:00 Aug22 08:00 Aug22 12:00 Aug22 16:00 Aug22 20:00 0 10 20 30 40 50 60 70 80 90

(26)

Memory Analysis

14 Aug 2000 08:00 - 20 Aug 08:00 HR: Work Mem Pct

0-Compiler 0-Database 0-Library 0-MCS

0-Other 0-Remote 0-ViewPoint 0-WFL

Aug14 08:00 Aug15 08:00 Aug16 08:00 Aug17 08:00 Aug18 08:00 Aug19 08:00 Aug20 08:00 0 5 10 15 20 25 30 35 40 45

(27)

I/O Anomaly

21 Au g 2000 20:00 - 22 Au g 20:03

F am Secs/IO fo r B CD APPL 0.113 F am Secs/IO fo r BC D C IS 0.113 F am Secs/IO fo r C PC U R R PK 0.113 F am Secs/IO fo r CPMASTPK 0.113 F am Secs/IO fo r MER C H PK 0.113 F am Secs/IO fo r PC ISPK 0.113 IOP K B /Sec 93700.000 Scale 0.113 0.113 0.113 0.113 0.113 0.113 93700.000 Au g 21 20:00 Au g 22 00:00 Au g 22 04:00 Au g 22 08:00 Au g 22 12:00 Au g 22 16:00 Au g 22 20:00 0.00 0.02 0.04 0.06 0.08 0.10

(28)

I/O Block Sizes

21 Aug 2000 20:00 - 22 Aug 20:03 A19: Fam KB/Block

BCDAPPL BCDCIS CPCURRPK

CPMASTPK MERCHPK PCISPK

Aug21 20:00 Aug22 00:00 Aug22 04:00 Aug22 08:00 Aug22 12:00 Aug22 16:00 Aug22 20:00

0 50 100 150 200 250 300 350

(29)

Disk I/O Transfer Rate

21 Aug 2000 20:00 - 22 Aug 20:03 A19: Fam KB/Sec

BCDAPPL BCDCIS CPCURRPK

CPMASTPK MERCHPK PCISPK

Aug21 20:00 Aug22 00:00 Aug22 04:00 Aug22 08:00 Aug22 12:00 Aug22 16:00 Aug22 20:00 0 5000 10000 15000 20000 25000 30000 35000 40000 45000

(30)

Disk Service by Type

0.000 0.005 0.010 0.015 0.020 0.025

(31)

Capacity Planning Objectives

! Determine resources needed to support

projected workload

! Verify desired performance / service levels

can be met

! Define explicit list of questions to answer to

(32)

Capacity Planning Levels

! Simple Projection

" Make Rule Of Thumb (ROT) assumptions for CPU,

I/O and memory loading to stay within “safe bounds”

" Use operational analysis techniques to estimate

" Must understand component limits

" Need monitor and analysis tools to validate

! Model Projection

" Define processing types to be analyzed

(33)

On Line Workload Growth

Transactions/Second 0 10 20 30 40 50 60 70 80 90 4/ 6/ 1997 7/ 6/ 1997 10/ 6/ 1997 1/ 6/ 1998 4/ 6/ 1998 7/ 6/ 1998 10/ 6/ 1998 1/ 6/ 1999 4/ 6/ 1999 7/ 6/ 1999 10/ 6/ 1999 1/ 6/ 2000 4/ 6/ 2000 7/ 6/ 2000 Tran/Sec Expon. (Tran/Sec)

(34)

Batch Workload Growth

Active Accounts vs Transaction Volume

0 10,000,000 20,000,000 30,000,000 40,000,000 50,000,000 Nov -97 Feb-9 8 Ma y-98 A ug-98 Nov -98 Feb-9 9 Ma y-99 Aug -99 Nov -99 Feb-0 0 M ay-00 Aug -00 No v-00 Ac ti v e Ac c o u n ts 0 2,000,000 4,000,000 6,000,000 8,000,000 10,000,000 Tr a n sa c ti o ns

(35)

CPU Usage vs Transactions

Weekly Transaction Rate

10 May 92 - 27 Feb 94 Average CPU Busy Percent

CPU BUSY Percent - Function of Transaction Rate

WK1: COMS Tran/Sec for All 0 10 20 30 40 50 60 70 80 90 100 0 1 2 3 4 5 6 7

(36)

CPU Workload Capacity

Projection

0.0% 20.0% 40.0% 60.0% 80.0% 100.0% 120.0% 140.0% 160.0% A 1 8 -6 3 2 C P U US A G E 1998 1999 2000 2001 2002 2003 Month - Year

APPLICATION CPU REQUIREMENT PROJECTION SIMPLE GROWTH

SALES OPERATIONS LAWSON PURCHASING CASH-N-CARRY SUPPORT WAREHOUSE MISC. PROD ORDER ENTRY ADMIN

INVENTORY CTRL PROMOTION DEVELOPMENT MCP

(37)

CPU Capacity Projection

CPU REQUIREMENT PROJECTION 25% LATENT DEMAND - Optimistic Growth

0% 50% 100% 150% 200% 250% 1998 1999 2000 2001 2002 2003 Month - Year CP U US AG E

CPU SERVICE LIMIT A18-632 USAGE NX5802-72 USAGE

(38)

Capacity Models

! Characterize the workloads (CPU, memory,

I/O, Delays)

! Define the hardware configuration

! Execute the model using current measured

data

! Validate against actual measurement

! Change the model workload levels – rerun

! Change the model parameters (HW/SW

spec) and rerun

(39)

A Workload Profile

MC0150 (Run 1) Transaction Profile 8 C opies MC0150 Running Concurrently

0.00000 0.00200 0.00400 0.00600 0.00800 0.01000 0.01200 Resource C omponent Res o u rce T im e p e r T ran sa c tio n Other Ti me 0.00115 IO Wait Ti me 0.00248

I/O Time/ Transaction 0.00323

R eadyQ per Transaction 0.00067

CPU Per Transaction 0.00280

(40)

Linear Projection from Model

0:00 1:00 2:00 3:00 4:00 5:00 6:00 5 ,5 74 ,8 63 6 ,9 68 ,5 79 8 ,3 62 ,2 95 9 ,7 56 ,0 10 11 ,1 49 ,7 26 12 ,5 43 ,4 42 13 ,9 37 ,1 58

Tot al Transactio n Growth

MC0150 (Run 1) Transaction Growth with Linear Projections (Elapsed Time)

(41)

Batch Model Results

0:00:00 2:00:00 4:00:00 6:00:00 8:00:00 10:00:00 12:00:00 14:00:00 16:00:00 18:00:00 20:00:00 22:00:00 24:00:00 H o u rs to C o m p le ti o n 28, 800 ,0 0 0 36, 000 ,0 0 0 43, 200 ,0 0 0 50, 400 ,0 0 0 57, 600 ,0 0 0 28, 800 ,0 0 0 36, 000 ,0 0 0 43, 200 ,0 0 0 50, 400 ,0 0 0 57, 600 ,0 0 0 28, 800 ,0 0 0 36, 000 ,0 0 0 43, 200 ,0 0 0 50, 400 ,0 0 0 57, 600 ,0 0 0 0% 25% 50% 75% 100% 0% 25% 50% 75% 100% 0% 25% 50% 75% 100% NX5822-710 NX5822-78 NX6826-PL10

Combined Simulation Modeling Projection for Critical Path Programs

8:51pm September 25th - 5:55am September 26, 2000 (1 Hour and 21 Minut es of Delay between Crit ical Path Programs)

Elapsed Time - MC0150 Elapsed Time - MC0182 Elapsed Time - MC0150-2 Elapsed Time - MC0338

Configuration Growth Scenario Total Accounts Component

(42)

Batch Cycle Table

CPU Secs ReadyQ Secs Other Delay Elapse IO Sec Pre-Batch Prep 5100 1083 0 11757

Critical Nightly Batch 10563 2425 614 16993

Post Nightly Batch 17920 1777 2928 26556

(43)

Batch Cycle Model

End of Month Batch Cycle Projections

(pro jectio ns include EM C 8350 IO upgrade)

0 5 10 15 20 25 30 Cu rren t N X48 22-2 2 NX -683 2-P L5 CS S78 017-2 8 CS S78 01 7-39 CS S78 01 7-52 NX 48 x2-3 2 NX 48 x2-4 2 NX 48 x2-5 2 Hours Elaps e IO Hours Othe r De lay Hours Re adyQ Hours CPU Hours

(44)

On-Line Workload Projection

Average Weekly Transaction Counts vs. Total Accounts

0 500000 1000000 1500000 2000000 2500000 3000000 3500000 4000000 1999 2000 2001 2002 2003 2004 2005 2006 Weekly COMS Transaction Count Total Accounts

(45)

Workload Projection

Consolidated Workloads Peak Composite (based on NX6832-PL5):

Dec-02 Dec-03 Dec-04 Dec-05 Dec-06 Dec-07

Sys-1 User Growth Rate 20% 28% 28% 11% 11% 11%

Aug-02 Dec-02 Dec-03 Dec-04 Dec-05 Dec-06 Dec-07

CPU MCP % 5.0 5.0 5.0 5.0 5.0 5.0 5.0

Sys-1 CPU User % 41.0 49.2 63.0 80.6 89.5 99.3 110.2

Sys-2 CPU User % 12.3 14.8 16.2 17.9 19.6 21.6 23.8

Sys-3 CPU User % 11.5 13.8 15.9 18.3 20.3 22.5 25.0

(46)

RPM Growth Projection

RPM Grow th Projections 0 1000 2000 3000 4000 5000 6000 Au g-02 Fe b-03 Au g-03 Fe b-04 Au g-04 Fe b-05 Au g-05 Fe b-06 Au g-06 Fe b-07 Au g-07 RPM RPM Requirements

(47)

RPM vs Capacity

RPM Requirements vs. System Capacity

0 1000 2000 3000 4000 5000 6000 Au g-02 Feb-03 Aug -03 Feb -04 Au g-04 Feb-05 Au g-05 Feb-06 Au g-06 Feb-07 Aug -07 RPM RPM Requirements CSS78017-39 Capacity NX6832-PL5 Capacity NX4822-22 Capacity

(48)

RPM Capacity Plan

RPM Requirements vs. Proposed System Capacity

0 1000 2000 3000 4000 5000 6000 7000 8000 Au g-02 Fe b-03 Au g-03 Fe b-04 Au g-04 Fe b-05 Au g-05 Fe b-06 Au g-06 Fe b-07 Au g-07 RPM RPM Requirements CSS78017-70 Capacity CSS78017-52 Capacity CSS78017-39 Capacity CSS78017-28 Capacity

(49)

IO Capacity

IO Bandpass Growth Considerations for an NX6836-PLA

Actual Capital One and Theoretical I/O Levels

Theoretical Capacity I/O Projections Based on Actual IO Blocksize and Estimated I/O Bandpass

0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 18,000 20,000 Processing Window I/ O s p e r sec o n d

NX6836-PLA I/O BandPass Million Bytes/Sec (Est) 101,351,250 102,945,107 103,762,817 111,834,557 108,997,543 Actual Avg IO Block in Bytes 11,700 17,100 14,700 18,900 17,800 Theoretical I/Os per Second 8,663 6,020 7,059 5,917 6,123 Actual Capital One IOs per Second Observed 5,515 3,947 4,695 4,447 4,425 Estimated % IORate Grow th over Observed Rates 57% 53% 50% 33% 38% GUI MC0150-1 MC0182 MC0102 MC0338

(50)

The Resulting Plan

! Install Libra Model 180 at capacity level

CSS78017-39 offering 3800 RPM useable Now.

! Purchase Capacity-on-Demand increments

as needed in 2004

! Upgrade to CSS78017-52 (5000 RPM

useable) in 2005 if growth projections hold into mid 2004.

(51)

In Conclusion

! IT Service Delivery has become increasingly

complex with the distribution of processing to multiple networked platforms.

! Fortunately, tools to monitor and manage

performance and service delivery of the entire IT infrastructure in real time are now available.

! Integrated support for capacity projection and

modeling is increasingly important to avoid service level degradation

(52)

Questions?

Thank you for your attention. Are there any questions?

(53)

DownLoad Available

! This presentation, complete with the

speakers notes and annotations will be available on the MGS, Inc web site

(www.mgsinc.com) for anonymous download

(54)

Contact Information

! MGS, Inc. is the software engineering

organization that builds the ViewPoint and SightLine products for A Series, ClearPath and ClearPath Plus (MCP).

! MGS is a distributor of the SightLine products

built by FORTEL Inc.

! Contact Guy Bonney

MGS, Inc. 10901 Trade Road Richmond, VA 23236 (804) 379-0230 www.mgsinc.com [email protected]