Perforce Load Testing - AgendaPerforce Load Testing - Agenda
• Speaker introduction
• Presentation scope
• Corporation background
• Load Test initiating: motivation, goals
• Load Test planning: software, hardware, licenses, clients
• Load Test execution: user emulation limits, escalation
• Load Test control: monitor clients and metrics
• Load Test closing: charts, analysis, lessons learned
Perforce Load Test – Speaker IntroductionPerforce Load Test – Speaker Introduction
• Dan Healy – daniel.f.healy@bankofamerica.com
– Current SCM admin based in London, UK
– Former SCM admin for BofA in San Francisco & Concord, CA
– Former index, currency and hedge fund application developer at Wells Fargo Investment Advisors and Barclays Global Investors in San Francisco
– Former Defence Dept. developer at White Sands, NM
– Academic background in physics, astronomy and computer science
Perforce Load Test – Presentation ScopePerforce Load Test – Presentation Scope
• Primary focus on why and how the load test was conducted
• Sterling test results should not be a surprise to anyone
• Provide a list of items to consider when planning, configuring and executing a Perforce load test
Perforce Load Test – Corporation BackgroundPerforce Load Test – Corporation Background
• Bank of America
– First/second largest financial institution in the world
– 200,000+ employees, 10,000 developers
– 21,000,000 online banking customers
– 443,000,000 ATM transactions per month
– 3.9 billion bankofamerica.com hits per month
– Pioneered consumer loans and branch banking
– Historical technology innovator
• First credit card – Bankamericard now known as Visa
• Magnetic ink numbering on checks
– Current technology innovations
• Distributed Computing
• Continuous Integration
Perforce Load Test - InitiatingPerforce Load Test - Initiating
• Application development highly dependant on globally
distributed teams
– ClearCase insufficient
– 3 separate rogue development teams began using Perforce
• Architecture Review Board (ARB) certification required
– Management skeptical regarding Perforce
• Perforce load test added to certification submission
– Determine any hardware or performance limitations on BofA hardware running extraordinary usage
Perforce Load Test - PlanningPerforce Load Test - Planning
• 2000 temp user licenses from Perforce
• Software
– Latest version of Perforce – 2006.1
– Perforce support load test engine
• Created a client workspace template called “bulk”
• Runs sync, flush, info & files, “x” iterations, “y” users, unique prefix for user IDs
– Server performance monitoring – RRDTool
– Control Panel Administrative Tools – Performance
– Excel spreadsheet
• Test servers
– similar setup to production environment
• Primary server in UK – Linux 4 CPU i686, 2.6.9 kernel
• Proxy servers in Chicago & Charlotte, Solaris 5.8, Sun Ultra-250
• 3 155MB T1’s between US & UK
• Test clients
– Cooperation from several colleagues to use their PCs
Perforce Load Test - ExecutingPerforce Load Test - Executing
• How many users can each PC emulate?
– PC disk space limitations vs. ample workspaces
– Settled on a 300MB depot and workspace
– 5-60 users per PC
• Used Control Panel Scheduled Tasks
– Typical activity: 20 users/50 iterations every 10 minutes
– Annoying pop up window
• Escalated usage to 800 users
• Occasional manual remove & submit
– 969 files, 8 folders, 56.2MB
Perforce Load Test – Monitoring & ControlPerforce Load Test – Monitoring & Control
• PC performance tool and graphics ensured clients were
not limiting activity
• Every morning, made sure all clients were active using
Perforce GUI
• Use RRDTool before and during major activity
– Number of connections
– Number of processes
– % CPU
– Load average
– Memory usage
– Manually recorded of number of users
• Determined production ClearCase and Perforce usage
• Determined server peak levels
– Memory
– % CPU
Perforce Load Test - ClosingPerforce Load Test - Closing
•
Analysis
– BofA hardware remains effective up to about 500
simultaneous intensive users
– Most vulnerable metric is memory
• Extrapolated peak at about 1000 users
•
Lessons learned
– More user participation and/or training rooms
– Employ UNIX/Linux clients
– Capture of PC performance graphics
– Automate peak activity
