Concurrent Class
10/3/2013 3:00:00 PM
"Decoupled System Interface
Testing at FedEx"
Presented by:
Chris Reites
FedEx
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
As a Technical Principal in software quality assurance for FedEx Services, Chris Reites has
experience providing cutting edge, best-practice processes for the development and testing of
large-scale, complex, global software systems. Chris has been working for FedEx in IT for
fifteen years. Prior to joining the software testing organization within FedEx, Chris was a
software developer for several key applications within the FedEx billing system.
Decoupled Testing
Overview
Chris Reites
Technical Principal – FedEx Services
STARWEST Conference
10/03/2013
FedEx Testing – What We’re Dealing With
•
Application Testing and Certification
•
Responsible for Test Planning,
Design, Execution, and Validation
•
Key Shipping Products (Desktop
devices,
www.fedex.com
, etc…)
•
Backend Rating, Revenue, Tracking,
and Invoicing Systems
•
Major releases annually, as well as
weekly exception and emergency
loads
Types of Testing We Do
3
Definition: Validates core software functionality in production after regularly scheduled software loads and regularly scheduled data updates. Supports Corporate Loads, Dotcom Loads, Exceptions Loads & some Emergency loads.Functional Testing
Performance Testing
Vulnerability Testing
Regional Testing
Definition: Evaluates the compliance of a system or component with specified functional requirements. Includes New Features, Regression, Integration and System tests. Coordinate with Revenue Testing for impacted products.
Definition: Evaluation of a system’s or component’s compliance with specified performance requirements. Includes Volume testing, Load / Stress testing, Failover / Recovery Testing and Disaster Recovery.
Definition: Utilizing security scanning tools and educated vulnerability testing through manual human intervention techniques, providing scanning and penetration testing for supported applications during regular releases.
Definition: Includes functional testing performed on behalf of the Regions by SQA Testing groups and Language Translation Testing performed by Marketing or User groups within the Regions.
Certification Services
Quality Assurance Approaches
Production Checkout
CSP Certification
Label Certification
(ensures operational excellence)
Definition: Functional tests designed to validate specific outcomes within Revenue systems. Starts with entering shipments (via INET, WSVC, CAFE, FXRS, etc.) and addition of selected scans, flows through servers and intermediate systems, and concludes with validation of results on invoices and in accounts receivables.
Definition: Consulting with third-party software providers to integrate FedEx technology into their applications and validate that their applications meet FedEx requirements from a brand, revenue and operational perspective.
Definition: Validation of Express labels submitted by automation clients, CSP providers, WebServices customers and FXRS customers who modify labels in production. Automation clients submit all labels for certification.
End-to-End Testing
•
Dependency to have all code ready at the same
time
•
Interfaces were critical…yet not well documented
or understood
•
Interface changes coming late into Code/Test
phases
•
Interface issues caused “Ripple Effect” throughout
system
Impacts our Speed To Market and causes a lack of
flexibility in testing
4
Potential Hazards in Large
System Testing
Decoupled Testing Concepts and Potential
5
Automation Revenue
Corp. Load End to End Testing
Corp. Load End to End Testing enhanced by Decoupled Test
Concept
•
Provides the ability to test target systems
independently by removing dependencies
on other external systems.
•
Divide and conquer
•
Reduce defect fix/validation cycle time
•
Mitigate Risk when introducing software
changes – comprehensive regression test
•
Reduce validation dependency on
End-to-End cycles
Adoption in Automation Systems
•
Reduced dependency on revenue test
cycles
•
Mini Revenue cycles within automation
systems using Decoupled Test Tool
•
Pilot mini Decoupled Testing cycle in one
shipping device – FY12 Q1
•
Mini Decoupled Testing cycles in multiple
shipping devices for Jan12 corporate
load.
Automation Revenue Devices Front End Process Back End Process Edit & Rating Invoicing Settlement Devices Front End Process Back End Process Edit & Rating Invoicing SettlementWhat Makes Up Decoupled Testing?
Decoupled Testing provides the ability to test target systems
independently by removing dependencies on other external
systems.
4 Core Functions:
•
Interface Data Capture
:
supports the collection and storage of interface
data
•
Interface Data Compare:
supports on-demand, field-level comparisons of
interface data
•
Interface Data Injection
:
supports the ability to ‘replay’ previously
captured interface data into the target system
•
Interface Simulation
:
supports the virtualization of responses from
eCommerce
Express Rev Rating
Ground Rev Rating
Accounting Corp App
Revenue Back End Regions Apps Invoice In p u ts O u tp u ts GL Entries A/R Order Cash Shipment Sys Testing Researcher Input Output Expected FCIS Sys
Freight Rev Rating
End to End
Corporate Load Testing
Coupled Testing
Jump
7
eCommerce
Express Rev Rating
Ground Rev Rating
Accounting Corp App
Revenue Back End Regions Apps Invoice In p u ts O u tp u ts GL Entries A/R Order Cash Shipment Sys Testing Researcher Input Output Expected FCIS Sys
Freight Rev Rating
Coupled Testing
Jump
Express Rev Rating Accounting Corp App Revenue Back End Invoice In p u ts O u tp u ts GL Entries A/R Order Cash Shipment Sys Testing Researcher Input Output Expected FCIS Sys Freight Rev Rating
End to End Testing
Cluster Cluster Cluster Cluster Cluster Cluster Cluster eCommerce Back End eCommerce Front End Cluster Cluster
Decoupled Testing
evolve
evolve
evolve
evolve
9
eCommerce
Front End
Express
Rev Rating
Revenue
Back End
In
pu
ts
Actual Before
Actual After
O
up
ut
s
Testing Researcher A A B B C SExpected
Decoupled Testing
10
11
Revenue Systems Example:
From Serial to Parallel Processing
12
•
New test data created for both system entry
points and downstream injection points
simultaneously
•
Regression test data injected with previously
captured results
•Compare outputs against previous results
Automation
•Inject data directly into backend system •Compare outputs against previous
results
Revenue
Common Test Data Design
Positive Effects:
Test Data
Shipping Input
Revenue Input
•
Reduce idle time (waiting for successful
end-to-end execution)
•
Increase the test coverage utilizing data
comparison analysis
•
Early detection of issues
Interface Simulation
13
•
Interface Simulation is
the ability to virtualize
responses from backend
systems.
•
Simulated Interfaces
remove backend
complexity from testing
environments and provide
stable, predictable
behavior to make system
testing easier and more
available.
•
Response times of virtual
interfaces can be varied to
simulate latency or
systems under load
Test Strategy Objectives
• From weeks to days or hours
Reduce Defect Resolution Time
• By providing data analysis solutions
• By automating validation processes
Reduce Validation Time
• By discovering and resolving defects earlier
• By more quickly fixing and revalidating defects
Improve Code Quality coming into Integration Testing
• By providing environments on demand
• By providing a self service set of test solutions
Common Goals And Success Metrics
Normal Dev/Test Phase DTT Dev/Test Phase
Agility and Time-to-Market
The measure of time required to launch a
project/feature through newly enabled testing
processes.
Defect Removal Cost Reduction
The measure of identifying and fixing defects
earlier in the testing process.
Support Resource Cost Reduction / Reallocation
The measure of resource reduction due to
functionality being delivered.
15
#
of
De
fe
ct
s
16
Defect Removal - Reduction in Cost
The measure of identifying defects earlier in the testing process.
Metrics above reveal the following:
•
A definite shift in time in the number of defects found and closed in earlier cycles
of testing
#
of
De
17
Decoupled Testing – Speed to Market
100.00%
67.00%
38.00%
0
0.2
0.4
0.6
0.8
1
1.2
Traditional
Decoupled Testing v1
Decoupled Testing v2
Dur
atio
n
of T
es
ting
R
elat
iv
e
To
T
raditio
n
al
Me
th
o
d
*
Testing Method
Testing Process for Certification of a Key Customer Shipping Platform
* Duration includes shipping/execution and validation
8.5%
100.0%
147.0%
100.0%
100.0%
147.0%
Traditional
Decoupled v1 Decoupled v2
#
Tes
t
Cas
es
Shipp
ed
an
d
V
al
id
at
ed
R
el
at
iv
e
to
Tr
ad
it
ion
al
Me
thod
Testing Method
Validated
Baseline
Decoupled Testing – Resource Cost Reduction
Testing Process for Certification of a Key Customer Shipping Platform
100%
1750%
3268%
Traditional Decoupled
v1
Decoupled
v2
%
Of
Tes
t
Cas
es
T
es
ted
P
er
Da
y
R
el
at
iv
e
to
Tr
ad
it
ion
al
Me
tho
d
Testing Method
•
For each test case: field level validation on
two transactions
•
Around 200 field comparisons per test case
100.00%
67.00%
38.00%
Traditional
Decoupled
Testing v1
Decoupled
Testing v2
Dur
ati
on
of
T
e
sti
n
g
R
e
la
ti
ve
To
Tr
ad
iti
on
al
M
eth
od
*
Testing Method
Case Study: FedEx Delivery Manager
19
FedEx Delivery Manager – Decoupled Testing
20
Business Challenge
Parallel development and
simultaneous delivery of
multiple FedEx
applications impacted by
the project prevented
integration testing prior
to Integration Testing.
Goals
Provide development
teams access to key
backend systems during
Unit Testing in order to
identify defects early,
and have the ability to
inject transactions into
various parts of the
system to break
dependency of needing
all systems ready at same
time.
Utilizing DTT, FedEx Delivery Manager early discovered 75% of defects and
validated 68% of test cases for the Back-End Systems
Delivery Manager Portal started having success
