International Conference On Software Testing Analysis & Review
T15
Thursday, October 30, 2003 3:00 PM
I
NCREASE THE
V
ALUE OF
Y
OUR
T
ESTING WITH
B
USINESS
-ORIENTED
M
ETRICS
Claude Fenner
Arsin Corporation
manager, quality analyst, and software designer. His experience includes enterprise application software, mission-critical system software, and commercial and military embedded software. Across these industries he has transformed organizations to deliver high-quality software at lower cost through automation and globalization
Arsin Corporation is a software reliability services and product company, specializing in enterprise testing, quality, and readiness assessment. Arsin provides expertise and tools to several Fortune 100 companies. Arsin also partners with major tool vendors such as Mercury Interactive and has a wholly-owned subsidiary Arsin Limited in India. Claude Fenner is
responsible for driving Arsin’s tools and processes into market-ready product solutions. During his career, he has driven large-scale quality and testing programs for major quality-oriented companies including Compaq, Tandem Computers, and SAIC. At Tandem, he built and led the quality teams that delivered Tandem’s high availability database and transaction
processing technologies used for stock exchanges and ATM networks. At Compaq he defined and drove the Y2K process that successfully updated Compaq’s more than 1000 NonStop branded products. Recent enterprise experience includes high volume B2B and B2C middleware for financial transactions. Beyond the enterprise arena, his experience includes developing processes and building tools for assuring the reliability of commercial nuclear safety systems and embedded military systems.
ARSIN
Corporation
Powering Collaborative e-Business
Increase the Value of your Testing
with Business-Oriented Metrics
Claude Fenner – StarWest 2003
A lot has changed in the last decade
Now
Brand
Then
Brand
Not everyone’s perspective on quality is exactly the same
Customer thinks: “I’ve paid for a service, so give me what you promised”
Service Quality
Process Quality
Customer
Producer
Business Owner
Producer thinks: “I’ve been told what customers want, so that’s what I’ll build”
Owner thinks: “I want customer satisfaction and I want to produce value for
business stakeholders”
Ok, you’re visible to business owners … Now what!
Bad News: Traditional metrics don’t resonate with business managers
Let’s take a closer look at a business owner
They say things like …
Are these tests really assets? Are they worth it? Why?
Can I squeeze more out of them? Can I get by with less?
Are things good enough for now? Can I wait later to invest?
What about assets needed for future demand?
Here are things they think about …
Are our customer being served?
What is the best mix of people, capital, geographical locations?
You’d better get ready to think like a business owner …
Customer Satisfaction
Threat/Risk to service quality
Business Value
TCO (Total Cost of Ownership) of testing
Test asset investment and potential asset gaps over time
Hint: Quality Theory is not the answer
Answer: Business-Oriented Metrics
Business-oriented metrics bridge the gap between the
activities your test organization performs and the
First, a little introduction to business-oriented metrics
The face of a business owner
accountable to stakeholders and customers.
Here are some metrics. Are they useful?
Predicted potential supply
Maximum supply
Actual demand
Predicted demand
These metrics serve the customer pretty well …
but not the producer or the business owner
How well do these metrics serve their audience?
Customer A+
Good for predicting daily outage risk and when outage might occur.
Producer
D-
Doesn’t indicate anything useful about the operation of a powerplant
(generation efficiency, power quality (brownouts), TTF)
Business Owner F
Too tactical: Only tells owner supply and demand for ONE DAY
Big gaps are uninformative:
Sense of false security. Disaster could strike without warning!
(unplanned failures, demand spike, plant decommissioning)
And look what happened
Lesson: Gray Davis didn’t see how increasing demand
was threatened by under-investment in assets and lack
of asset health.
How to transform these into business-oriented metrics …
coal
hydro
oil
nuclear
oil
4. Indicate future demand
for assets
3. Add time (past and future)
1. Show assets individually
2. Indicate asset growth and
decay (health)
time
But don’t forget, business owners also care about value
Efficiently run businesses run close to the edge.
Business-Oriented metrics in the Testing World
Before starting, remember what you’re trying to do
Talk to business owners in terms they understand
Influence decisions by demonstrating and defending testing value
Get them to participate (be a stakeholder) in your process
Compete against others for scarce resources and win
Always remember the basics of metrics: It’s a sequence of
steps from objectives to actions.
1.Define Objectives and Metrics
2.Collect Data
3.Reveal Trends
4.Make Decisions
5.Take Actions
Business-Oriented testing metrics defined
Test Asset Trajectory metric (benefit-side)
Identifies testing assets and measures how adequate they are at
meeting testing demand. Also shows asset benefit, health, and risk.
The term trajectory is used to indicate that the metric is always
measured over time.
Test Asset Leverage metric (cost-side)
Identifies the costs associated with the testing assets. The term
leverage is used to indicate that the metric subdivides cost into
components that are useful for identifying best utilization of
resources.
Together they form the total asset value picture.
asset-value = benefit-side + cost-side
Test Asset Trajectory metric
Manual Automated Functional Stress Demand End-to-end Performance Manual Automated FunctionalAssets & Types
Automated Functional Performance Manual End-to-end End-to-end Performance Manual Automated Functional Automated Manual Stress Today Manual Performance Automated Functional TIme Timeframe & Events
Completeness & Health
Manual Automated Functional Stress End-to-end Performance Manual Automated
Functional AutomatedFunctional Performance Manual End-to-end End-to-end Performance Manual Automated Functional Automated Manual Stress Manual Performance Automated Functional
Test Asset Leverage metric
End-to-end Performance Manual Automated Functional End-to-end Performance Manual Automated Functional Automated Manual Stress Automated Functional Performance Manual End-to-end Performance Automated Functional TIme Automated Functional Stress Manual3
5
3
People Machines Dollars Time People Machines Dollars Time People Machines Dollars Time People Machines Dollars TimeSingle, clear statement of the asset cost by component
Asset Cost by component Assets Total Cost
=
11
+
+
Test Asset metrics building block
Asset cost (in all-or-nothing chunks)
# People (by geography, by FTE/Contractor, Outsourcer)
# Machines (clusters, labs, in some cases single units)
Dollars for tools or testing supply Asset benefit
%completed (degree to which demand is met by asset)
%healthy (how much of asset is fully fit for service) Asset Type Definition
Name: manual, performance, automated, etc Demand
(e.g. function points, list of test items, test scenarios)
Every building block contains the same information. Demand,
asset trajectory and asset leverage metrics are computed by
aggregating the information from all the building blocks.
Case Study – Software Product Company
Profile
Web services product for handling B2C and B2B financial transactions.
Major business events related to Testing
New Functionality
Functional Tests
Rapid Adoption
Scalability & Performance
Technology Changes
Regression
Acquisition
Geographical Distribution
Automation Manual
Test
Conversion
Customer-driven Releases
Rapid Regression
Workforce Redistribution
Employees, Contractors, Outsource
Actions and Results
Team transformed from handling a single product line to three product
lines with 75% less people! Scalability for multiple thousands of users
proven. Outsourcing option enabled.
Case Study – Test Asset Trajectory (3 products 2 years)
Demand
Assets & Types Timeframe & Events
Completeness & Health
TIme Pro duct A rele ase Pro duct A do t relea ses Pro duct A scala blity 5k Pro duct A scala blity 10 k Pro duct C rele ase Pro duct B ac quis ition Tec hno logy c hange Fin anc ial m isfortu ne Wo rkfo rce reorg Ou tso urc ing Stress Manual Automated Functional Performance Manual End-to-end Automated Functional Performance Manual End-to-end Automated Functional Performance Manual Automated Functional Performance Manual Automated Functional Performance Manual
A
Manual Manual Automated Functional Manual Automated Functional Manual Automated FunctionalB
Automated Functional Performance Manual Automated Functional Performance ManualC
Stress Manual Automated Functional Performance Manual End-to-end Automated Functional Performance Manual End-to-end Automated Functional Performance Manual Automated Functional Performance Manual Automated Functional Performance Manual Manual Manual Automated Functional Manual Automated Functional Manual Automated Functional Automated Functional Performance Manual Automated Functional Performance ManualStart: 8 people for 1 product End: 5 people for 3 products Due to automation investment
Case Study – Test Asset Leverage (3 products 2 years)
TIme Stress Manual Automated Functional Performance Manual End-to-end Automated Functional Performance Manual End-to-end Automated Functional Performance Manual Automated Functional Performance Manual Automated Functional Performance Manual
