Business Value of. Agile Testing. Using TDD, CI, CD & DevOps. Dr. David F. Rico, PMP, CSEP, FCP, FCT, ACP, CSM, SAFe

(1)

Business Value of

Agile Testing

Using TDD, CI, CD & DevOps

Dr. David F. Rico,

PMP

,

CSEP

,

FCP

,

FCT

,

ACP

,

CSM

,

SAFe

Twitter

:

@dr_david_f_rico

Website:

http://www.davidfrico.com

:

http://www.linkedin.com/in/davidfrico

Agile Capabilities

:

http://davidfrico.com/rico-capability-agile.pdf

Agile Resources

:

http://www.davidfrico.com/daves-agile-resources.htm

(2)

Author Background



Gov’t contractor with

33+ years

of

IT experience



B.S.

Comp

.

Sci

., M.S.

Soft

.

Eng

., & D.M.

Info

.

Sys

.



Large gov’t projects

in U.S., Far/Mid-East, & Europe

2





Career systems & software engineering methodologist



Lean-Agile, Six Sigma, CMMI, ISO 9001, DoD 5000



NASA, USAF, Navy, Army, DISA, & DARPA projects



Published seven books & numerous journal articles



Intn’l keynote speaker, 150 talks to 12,000+ people



Specializes in metrics, models, & cost engineering



Cloud Computing, SOA, Web Services, FOSS, etc.

(3)

Today’s Whirlwind Environment

3



Overruns



Attrition



Escalation



Runaways



Cancellation

Global

Competition

Demanding

Customers

Organization

Downsizing

System

Complexity

Technology

Change

Vague

Requirements

Work Life

Imbalance



Inefficiency



High O&M



Lower DoQ



Vulnerable



N-M Breach

Reduced

IT Budgets

81 Month

Cycle Times

Redundant

Data Centers

Lack of

Interoperability

Poor

IT Security

Overburdening

Legacy Systems

Obsolete

Technology & Skills

Pine, B. J. (1993). Mass customization: The new frontier in business competition. Boston, MA: Harvard Business School Press.

Pontius, R. W. (2012). Acquisition of IT: Improving efficiency and effectiveness in IT acquisition in the DoD. Second Annual AFEI/NDIA Conference on Agile in DoD, Springfield, VA, USA.

(4)

Software in U.S. DoD Systems

Kennedy, M. P., & Umphress, D. A. (2011). An agile systems engineering process: The missing link. Crosstalk, 24(3), 16-20.



No. of software-intensive systems is

growing



80%

of US DoD

functions

performed in software



Major driver

of

cost

,

schedule

, &

tech

.

performance

4

(5)

Software in U.S. DoD Avionics

Blackburn, M. R. (2014). Transforming systems engineering through a holistic approach to model centric engineering. Washington, DC: Stevens Institute of Technology.



Software in U.S. DoD avionics growing

exponentially



10x

growth from F-16 to F-22 (& another

10x

to F-35)



Productivity

must grow by

10x

for

next gen

systems

5

(6)

Requirements Defects & Waste

6

Sheldon, F. T. et al. (1992). Reliability measurement: From theory to practice. IEEE Software, 9(4), 13-20

Johnson, J. (2002). ROI: It's your job. Extreme Programming 2002 Conference, Alghero, Sardinia, Italy.



Requirements defects

are #1 reason projects fail



Traditional projects specify

too many requirements



More than

65%

of

requirements

are

never used

at all

Other 7%

Requirements

47%

Design

28%

Implementation

18%

Defects

Always 7%

Often 13%

Sometimes

16%

Rarely

19%

Never

45%

Waste



(7)

Traditional Projects

7



Big projects

result in

poor quality

and

scope changes



Productivity declines

with

long queues

/

wait times



Large projects

are

unsuccessful

or

canceled

Jones, C. (1991). Applied software measurement: Assuring productivity and quality. New York, NY: McGraw-Hill.

Size vs. Quality

DEFECTS

0.00 3.20 6.40 9.60 12.80 16.00 0 2 6 25 100 400

SIZE

Size vs. Productivity

PR

ODUCTIVITY

0.00 1.00 2.00 3.00 4.00 5.00 0 2 6 25 100 400

SIZE

Size vs. Change

CHANGE

0% 8% 16% 24% 32% 40% 0 2 6 25 100 400

SIZE

Size vs. Success

SUCCESS

0% 12% 24% 36% 48% 60% 0 2 6 25 100 400

SIZE



(8)

Global Project Failures

8

Standish Group. (2015). Chaos summary 2015. Boston, MA: Author.

Sessions, R. (2009). The IT complexity crisis: Danger and opportunity. Houston, TX: Object Watch.



Challenged

and

failed

projects hover at

67%



Big projects fail more often

, which is 5% to 10%



Of

$1.7T spent

on IT projects,

over $858B were lost

$0.0 $0.4 $0.7 $1.1 $1.4 $1.8 2002 2003 2004 2005 2006 2007 2008 2009 2010

Trillions (US Dollars)

Expenditures

Failed Investments

0% 20% 40% 60% 80% 100%

28%

34%

29%

35%

32%

33%

27%

28%

29%

49%

51%

53%

46%

44%

41%

56%

55%

52%

23%

15%

18%

19%

24%

26%

17%

19%

2000 2002 2004 2006 2008 2010 2012 2014 2015

Ye

ar

Successful

Challenged

Failed

(9)

What is Agility?



A-gil-i-ty

(ә-'ji-lә-tē) Property consisting of

quickness

,

lightness

, and

ease of movement

;

To be very nimble



The ability to create and

respond to change

in order to

profit in a turbulent global business environment



The ability to

quickly reprioritize

use of resources when

requirements, technology, and knowledge shift



A very

fast response

to sudden market changes and

emerging threats by intensive

customer interaction



Use of

evolutionary

,

incremental

, and

iterative

delivery

to converge on an optimal customer solution



Maximizing

BUSINESS VALUE

with right sized,

just-enough, and just-in-time processes and documentation

Highsmith, J. A. (2002). Agile software development ecosystems. Boston, MA: Addison-Wesley.

9

(10)

What are Agile Methods?

10



People-centric

way to create innovative solutions



Product-centric

alternative to documents/process



Market-centric

model to maximize business value

Agile Manifesto. (2001). Manifesto for agile software development. Retrieved September 3, 2008, from http://www.agilemanifesto.org

Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.

Rico, D. F. (2012). Agile conceptual model. Retrieved February 6, 2012, from http://davidfrico.com/agile-concept-model-1.pdf

Customer Collaboration

Working Systems & Software

Individuals & Interactions

Responding to Change

valued

more than

valued

more than

valued

more than

valued

more than

Contracts

Documentation

Processes

Project Plans



Frequent comm.



Close proximity



Regular meetings



Multiple comm. channels



Frequent feedback



Relationship strength



Leadership



Boundaries



Empowerment



Competence



Structure



Manageability/Motivation



Clear objectives



Small/feasible scope



Acceptance criteria



Timeboxed iterations



Valid operational results



Regular cadence/intervals



Org. flexibility



Mgt. flexibility



Process flexibility



System flexibility



Technology flexibility



Infrastructure flexibility



Contract compliance



Contract deliverables



Contract change orders



Lifecycle compliance



Process Maturity Level



Regulatory compliance



Document deliveries



Document comments



Document compliance



Cost Compliance



Scope Compliance



Schedule Compliance



Courage



(11)

Agile World View



“Agility” has many

dimensions

other than IT



It ranges from

leadership

to

technological

agility



Today’s focus is on

organizational

&

enterprise

agility





Agile Leaders

Agile Organization Change

Agile Acquisition & Contracting

Agile Strategic Planning

Agile Capability Analysis

Agile Program Management

Agile Tech.

Agile Information Systems

Agile Tools

Agile Processes & Practices

Agile Systems Development

Agile Project Management

11

(12)

Network

Computer

Operating System

Middleware

Applications

APIs

GUI

How Agile Works



Agile

requirements implemented

in

slices vs. layers



User needs

with

higher business value

are done first



Reduces

cost & risk

while increasing

business success

12

Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway.

Agile

Traditional

1

2 3



Faster



Early ROI



Lower Costs



Fewer Defects



Manageable Risk



Better Performance



Smaller Attack Surface

Late



No Value



Cost Overruns



Very Poor Quality



Uncontrollable Risk



Slowest Performance



More Security Incidents



Seven Wastes 1. Rework 2. Motion 3. Waiting 4. Inventory 5. Transportation 6. Overprocessing 7. Overproduction

MINIMIZES

MAXIMIZES

 JIT, Just-enough architecture

 Early, in-process system V&V

 Fast continuous improvement

 Scalable to systems of systems

 Maximizes successful outcomes

 Myth of perfect architecture

 Late big-bang integration tests

 Year long improvement cycles

 Breaks down on large projects

 Undermines business success

(13)

Thousands of Tests

Continuously Executed

No More Late Big

Bang Integration



User needs designed & developed

one-at-a-time



Changes

automatically detected

, built, and tested



System

fully tested

and

deployed

as changes occur

13

Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.

Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley. Build Integration Server Version Control Server Build Scripts Uses Watches Build Status Provides Developer A Developer B Developer C Commits Changes Commits Changes Commits Changes

Builds

Database

Analysis

Testing

Reporting

Documentation

Deployment

Early, Automated, Fast,

Efficient, & Repeatable

Constant Readiness

State & CM Control

Lean, Waste Free, Low WIP,

No Deadlocked Test Queues

Rapidly & Successfully

Dev. Complex Systems

Basic Agile Mechanics

(14)

14



Methods to “scope” project, product, or system



“Key” is smallest possible scope with highest value



Reduces

cost

,

risk

,

time

,

failure

, &

tech

.

obsolescence

Barely Sufficient Design

M

INIMUM

M

ARKETABLE

F

EATURE

MMF

-

Advantage



Differentiator



Revenue



Profit



Savings



Brand



Loyalty

M

INIMUM

V

IABLE

P

RODUCT

MVP

-

Primary Goal



Main Process



Feature List



Priority Order



Story Map



System Design

S

TORY

M

AP OR

I

MPACT

M

AP

SM or IM

-

Goal



Actors



Impacts



Deliverables



Measurements



Milestones

V

ISION

S

TATEMENT

VS

-

For

<

customer

>



Who

<

needs it

>



The

<

product

>



Is a

<

benefit

>



That

<

customer

>



Unlike

<

other

>



Ours

<

different

>

I

NCREASES

T

ESTABILITY

, Q

UALITY

, R

ELIABILITY

, S

ECURITY

, M

ORALE

, M

AINTAINABILITY

, & S

UCCESS

Denne, M., & Cleland-Huang, J. (2004). Software by numbers: Low-risk, high-return development. Santa Clara, CA: Sun Microsystems.

Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation. New York, NY: Crown Publishing.

Patton, J. (2014). User story mapping: Discover the whole story, build the right product. Sebastopol, CA: O'Reilly Media.

Layton, M. C., & Maurer, R. (2011). Agile project management for dummies. Hoboken, NJ: Wiley Publishing.

(15)

15

Capability #1 ●Feature 1 ●Feature 2 ●Feature 3 ●Feature 4 ●Feature 5 ●Feature 6 ●Feature 7 Capability #2 ●Feature 8 ●Feature 9 ●Feature 10 ●Feature 11 ●Feature 12 ●Feature 13 ●Feature 14 Capability #3 ●Feature 15 ●Feature 16 ●Feature 17 ●Feature 18 ●Feature 19 ●Feature 20 ●Feature 21 Capability #4 ●Feature 22 ●Feature 23 ●Feature 24 ●Feature 25 ●Feature 26 ●Feature 27 ●Feature 28 Capability #5 ●Feature 29 ●Feature 30 ●Feature 31 ●Feature 32 ●Feature 33 ●Feature 34 ●Feature 35 Capability #6 ●Feature 36 ●Feature 37 ●Feature 38 ●Feature 39 ●Feature 40 ●Feature 41 ●Feature 42 Capability #7 ●Feature 43 ●Feature 44 ●Feature 45 ●Feature 46 ●Feature 47 ●Feature 48 ●Feature 49

1

2

3

4

5

6

7

8

9

10

11 12

13

14 15

16

17 18

19

20 21

Evolving “Unified/Integrated” Enterprise Data Model

“

Disparate

” L

EGACY

S

YSTEM

D

ATABASES

(

AND

D

ATA

M

ODELS

)

ETL A A B C D E F G H I J K A B C D E F A B C D E A B C D A B C A B

“Legacy” MS SQL Server Stovepipes “Inter-Departmental” Linux Blade/Oracle/Java/WebSphere Server

“Leased” DWA/HPC/Cloud Services

Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6 Sprint 7

ETL ETL ETL ETL ETL ETL

Bente, S., Bombosch, U., & Langade, S. (2012). Collaborative enterprise architecture: Enriching EA with lean, agile, and enterprise 2.0 practices. Waltham, MA: Elsevier.

(

for example, assume 25 user stories per feature, 175 user stories per capability, and 1,225 user stories total

)



Organize needs into capabilities, features, and stories



Prioritize features, group releases, and initiate sprints



Develop

minimum set of features

with

highest value

Agile Systems Development

Release Release Release Release



MMF or -MVP



(16)

Models of

A

GILE

D

EVELOPMENT

16



Agile methods spunoff flexible manufacturing 1990s



Extreme Programming (XP) swept the globe by 2002



Today

, over

90% of IT projects

use

Scrum/XP hybrid



Use Cases



Domain Model



Object Oriented



Iterative Dev.



Risk Planning



Info. Radiators



Planning Poker



Product Backlog



Sprint Backlog



2-4 Week Spring



Daily Standup



Sprint Demo



Feasibility



Business Study



Func. Iteration



Design Iteration



Implementation



Testing



Domain Model



Feature List



Object Oriented



Iterative Dev.



Code Inspection



Testing



Release Plans



User Stories



Pair Programmer



Iterative Dev.



Test First Dev.



Onsite Customer

Cockburn, A. (2002). Agile software development. Boston, MA: Addison-Wesley.

Schwaber, K., & Beedle, M. (2001). Agile software development with scrum. Upper Saddle River, NJ: Prentice-Hall.

Stapleton, J. (1997). DSDM: A framework for business centered development. Harlow, England: Addison-Wesley.

Palmer, S. R., & Felsing, J. M. (2002). A practical guide to feature driven development. Upper Saddle River, NJ: Prentice-Hall.

Beck, K. (2000). Extreme programming explained: Embrace change. Reading, MA: Addison-Wesley.

C

RYSTAL

M

ETHODS

-

1991

-

S

1993

CRUM

-

DSDM

1993

-

1997

FDD

-

1998

XP

-

Reflection W/S



Retrospective



Quality Control



Quality Control



Continuous Del.

(17)

Basic

S

CRUM

Framework

Schwaber, K., & Beedle, M. (2001). Agile software development with scrum. Upper Saddle River, NJ: Prentice-Hall.



Created by Jeff Sutherland at Easel in 1993



Product backlog comprised of prioritized features



Iterative

sprint-to-sprint

,

adaptive

&

emergent

model

17

(18)

Models of

A

GILE

P

ROJECT

M

GT

.

18



Dozens of Agile project management models emerged



Many stem from principles of Extreme Programming



Vision

,

releases

, &

iterative development

common



Prioritization



Feasibility



Planning



Tracking



Reporting



Review



Visionate



Speculate



Innovate



Re-Evaluate



Disseminate



Terminate



Scoping



Planning



Feasibility



Cyclical Dev.



Checkpoint



Review



Envision



Speculate



Explore



Iterate



Launch



Close



Vision



Roadmap



Release Plan



Sprint Plan



Daily Scrum



Retrospective

Thomsett, R. (2002). Radical project management. Upper Saddle River, NJ: Prentice-Hall.

DeCarlo, D. (2004). Extreme project management: Using leadership, principles, and tools to deliver value in the face of volatility. San Francisco, CA: Jossey-Bass.

Wysocki, R.F. (2010). Adaptive project framework: Managing complexity in the face of uncertainty. Boston, MA: Pearson Education.

Highsmith, J. A. (2010). Agile project management: Creating innovative products. Boston, MA: Pearson Education.

R

ADICAL

-

2002

-

E

2004

XTREME

-

A

2010

DAPTIVE

-

2010

A

GILE

-

S

-

IMPLIFIED

2011

(19)



Created by Mark Layton at PlatinumEdge in 2012



Mix of new product development, XP, and Scrum



Simplified

codification

of

XP

and

Scrum

hybrid

19

Simplified

A

GILE

P

ROJECT

M

GT

.

(20)

20



Numerous models of agile portfolio mgt. emerging



Based on lean-kanban, release planning, and Scrum



Include

organization

,

program

, &

project

management

Schwaber, K. (2007). The enterprise and scrum. Redmond, WA: Microsoft Press.

Leffingwell, D. (2007). Scaling software agility: Best practices for large enterprises. Boston, MA: Pearson Education.

Larman, C., & Vodde, B. (2008). Scaling lean and agile development: Thinking and organizational tools for large-scale scrum. Boston, MA: Addison-Wesley.

Ambler, S. W., & Lines, M. (2012). Disciplined agile delivery: A practitioner's guide to agile software delivery in the enterprise. Boston, MA: Pearson Education.

Thompson, K. (2013). cPrime’s R.A.G.E. is unleashed: Agile leaders rejoice! Retrieved March 28, 2014, from http://www.cprime.com/tag/agile-governance

Schwaber, K. (2015). The definitive guide to nexus: The exoskeleton of scaled scrum development. Lexington, MA: Scrum.Org

Models of

A

GILE

P

ORTFOLIO

M

GT

.

E

S

CRUM

-

2007

-

2007

SAFe

-

2007

L

E

SS

-

2012

D

A

D

-

2013

RAGE

-

2015

SPS

-

Product Mgt



Program Mgt



Project Mgt



Process Mgt



Business Mgt



Market Mgt



Strategic Mgt



Portfolio Mgt



Program Mgt



Team Mgt



Quality Mgt



Delivery Mgt



Business Mgt



Portfolio Mgt



Product Mgt



Area Mgt



Sprint Mgt



Release Mgt



Business Mgt



Portfolio Mgt



Inception



Construction



Iterations



Transition



Business



Governance



Portfolio



Program



Project



Delivery



Product Mgt



Program Mgt



Sprint Mgt



Team Mgt.



Integ Mgt.



Release Mgt



(21)

Scaled Agile Framework (

SAF

E

)



Created by Dean Leffingwell of Rally in 2007



Knowledge to scale agile practices to enterprise



Hybrid

of

Kanban

,

XP release planning

, and

Scrum

21

Leffingwell, D. (2007). Scaling software agility: Best practices for large enterprises. Boston, MA: Pearson Education.

(22)

22

Agile Performance Measurement

Work (Story , Point, Task) or Effort (Week, Day , H our)

Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)

Burndown

Cumulative Flow

Earned Value Management - EVM

CPI SPI PPC APC Work (Story , Point, Task) or Effort (Week, Day , H our)

(23)

What is Agile Testing?



Traditional testing is a late, manual process



Agile testing is an early and automated process



Goal to

deliver early & often

and

V&V components

23

Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from http://davidfrico.com/agile-testing-resources.txt

Crispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.

Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.

A

GILE

T

ESTING

-

Early Incremental Testing

-T

RADITIONAL

T

ESTING

-

Late Big Bang Integration Testing

-Test Criteria Accompany Stories



Automated Tests Written First



Units Coded-Tested One at Time



Code is Frequently Checked In



Code Automatically Retrieved



Code Automatically Compiled



Tests Automatically Executed



Instant Feedback & Test Reports

Test Criteria Written After Fact



Manual Tests Written Much Later



Units Coded Late All at One Time



Code Checked In Late in Project



Code Manually Submitted to Test



Code Manually Compiled & Built



Tests Manually Executed Late



Late Project Feedback & Reports



Code Automatically Deployed



Late Defects Freeze Projects

(24)

B

ASIC

—Test Driven Development



Term coined by Kent Beck in 2003



Consists of writing all tests before design



Ensures all components are verified and validated

24

Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.

(25)

A

DVANCED

—Continuous Integration



Term coined by Martin Fowler in 1998



Process of automated build/regression testing



Evaluates impact of changes against entire system

25

Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.



A

LL DEVELOPERS RUN PRIVATE BUILDS



D

EVELOPERS COMMIT CODE TO VERSION CONTROL



I

NTEGRATION BUILDS OCCUR SEVERAL TIMES PER DAY



100%

OF SYSTEM TESTS MUST PASS FOR EVERY BUILD



A

SHIPPABLE PRODUCT RESULTS FROM EVERY BUILD



F

IXING BROKEN BUILDS IS OF THE HIGHEST PRIORITY



R

EPORTS AUTOMATICALLY GENERATED

&

REVIEWED

(26)



Agile testing consists of seven broad practices



Automated

build

,

database

,

inspection

,

tests

, etc.



Include

reporting

,

documentation

,

deployment

, etc.

26

Practice

Building

Database

Inspections

Testing

Feedback

Documentation

Deployment

Description

Frequently assembling products and services to ensure delivery readiness

Frequently generating/analyzing database schemas, queries, and forms

Frequently performing automated static analysis of product/service quality

Frequently performing automated dynamic product and service evaluation

Frequently generating automated status reports/messages for all stakeholders

Frequently performing automated technical/customer document generation

Frequently performing automated delivery of products/services to end users

P

RACTICES

—Continuous Integration

(27)



Created by Jez Humble of ThoughtWorks in 2011



Includes CM, build, testing, integration, release, etc.



Goal is

one-touch

automation of

deployment pipeline

27

Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.

Ohara, D. (2012). Continuous delivery and the world of devops. San Francisco, CA: GigaOM Pro.



CoQ

• 80% MS Tst • 8/10 No Val • $24B in 90s • Rep by CD • Not Add MLK

E

NTERPRISE

—Continuous Delivery

(28)



Agile CD consists of seven broad practices



Automated

acceptance

,

load

,

performance

, etc.



Include

packaging

,

pre-production test

,

C&A

, etc.

28

Practice

Packaging

Acceptance

Load Test

Performance

Pre-Production

Certification

Deployment

Description

Frequently generating system images for pre-production testing & checkout

Frequently performing automated system & user acceptance testing

Frequently performing automated system load, stress, & capacity testing

Frequently performing automated system user & technical performance testing

Frequently performing automated pre-production tests prior to final deployment

Frequently performing automated system certification & accreditation tests

Frequently generating product images for pre-deployment testing & checkout

Mukherjee, J. (2015). Continuous delivery pipeline: Where does it choke. Charleston, SC: CreateSpace.

Swartout, P. (2014). Continuous delivery and devops: A quickstart guide. Birmingham, UK: Packt Publishing.

P

RACTICES

—Continuous Delivery

(29)



Created by Patrick Debois of Jedi BVBA in 2007



Collaboration of developers & infrastructure people



Goal to

automate

the

deployment

to

end-user

devices

29

Bass, L., Weber, I., & Zhu, L. (2015). Devops: A software architect's perspective. Old Tappan, NJ: Pearson Education.

Gruver, G., & Mouser, T. (2015). Leading the transformation: Applying agile and devops at scale. Portland, OR: IT Revolution Press.

Humble, J., Molesky, J., & O'Reilly, B. (2015). Lean enterprise: How high performance organizations innovate at scale. Sebastopol, CA: O'Reilly Media.

G

LOBAL

—Development Operations



(30)



Agile DevOps consists of seven broad practices



Automated

sys admin

,

CM

,

building

,

monitor

, etc.



Include

virtualization

,

containerize

,

deployment

, etc.

30

Practice

Sys Admin

Config. Mgt.

Host Builds

Virtualization

Containerize

Deployment

Monitor & Supp

Description

Frequently performing automated system administration tasks, i.e., scripting

Frequently performing automated infrastructure config. mgt./version control

Frequently performing automated system and server host builds and config.

Frequently performing automated system, server, & net virtualization services

Frequently performing automated software and Microservices containerization

Frequently generating final end-user system & software images for distribution

Frequently performing automated metrics collection & deployment monitoring

Duffy, M. (2015). Devops automation cookbook: Over 120 recipes coverying key automation techniques. Birmingham, UK: Packt Publishing.

Farcic, V. (2016). The devops 2.0 toolkit: Automating the continuous deployment pipelines with containerized microservices. Victoria, CA: LeanPub.

G

LOBAL

—Development Operations

(31)



Agile methods are based on

traditional measures



Story points

,

velocity

, and

burndown

basic metrics



Experts use

Agile EVM

,

test

,

ROI

&

portfolio metrics

31

Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.

A

GILE

M

ETRICS

1.Agile CODEMetrics 2.Agile PROJECTMetrics 3.Agile TRACKINGMetrics 4.Agile TESTINGMetrics 5.Agile VALUEMetrics 6.Agile HEALTHMetrics 7.Agile PORTFOLIOMetrics

1. Agile

C

ODEMetrics



Code Size



Code Complexity



Object Oriented



Code Coverage



Code Defects



Relational Design

2. Agile

P

ROJECT Metrics



Software Size



Software Productivity



Software Effort



Software Quality



Software Schedule



Software Success

3. Agile

T

RACKING Metrics



Story Points



Sprint Burndown



Release Burndown



Velocity



Feature Progress



Agile Earned Value

4. Agile

T

ESTINGMetrics



Test Coverage



Test Automation



Integration Builds



Running Tested Features



DevOps Automation



Deployment Frequency

7. Agile

P

ORTFOLIO Metrics



Portfolio Kanban



Epic Progress



Portfolio Radar



Release Train Radar



Lean Portfolio Metrics



Enterprise Scorecard

6. Agile

H

EALTH Metrics



Teamwork Quality



Collaboration Quality



Agile Process Maturity



Agile Adoption Rate



Degree of Agility



Product Flexibility

5. Agile

V

ALUE Metrics



Total Lifecycle Costs



Total Lifecycle Benefits



Benefit to Cost Ratio



Return on Investment



Net Present Value



Real Options Analysis

Agile

Testing

Metrics—Taxonomy

(32)

32

M

ETRIC

D

ESCRIPTION

T

EST

C

OVERAGE

Percent or degree to which software source code is tested

T

EST

A

UTOMATION

Ratio or degree to which software tests are automated

I

NTEGRATION

B

UILDS

Frequency of automated software builds and integrations

R

UNNING

T

ESTED

F

EATURES

Number of completed and tested features or user stories

D

EV

O

PS

A

UTOMATION

Ratio or degree to which deployments are automated

D

EPLOYMENT

F

REQUENCY

Frequency of automated software deployments or deliveries



Software test automation emerged during the 1970s



Reached their height in personal computer (PC) era



Most are

FOSS

and used by

successful

agile teams

Agile

Testing

Metrics—Definitions

(33)

Agile

Testing

Metrics—Example

33

(34)

34

Traditional vs. Agile Cumulative Flow

Traditional Cumulative Flow Agile Cumulative Flow



Late big bang integration increases WIP backlog



Agile testing early and often reduces WIP backlog



Improves

workflow

and

reduces WIP

&

lead times

Anderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education.

Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.

Agile

Testing

—Workflow

(35)



Fewer integrations leave in higher bug counts



Frequent, early integrations eliminate most defects



Goal is to have as many

early

integrations

as

possible

35

Lacoste, F. J. (2009). Killing the gatekeeper: Introducing a continuous integration system. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 387-392.





Number of

Integrations

Less Defects

•

More Integrations

•

Early Integrations

More Defects

•

Few Integrations

•

Late Integrations

Agile

Testing

—Economic Drivers

(36)

Tests

Hour

Day

Week

Month

Quarter

6 Months

Year

One

6

48

240

1,040

3,120

6,240

12,480

Three

18

144

720

3,120

9,360

18,720

37,440

Six

36

288

1,440

6,240

18,720

37,440

74,880

Twelve

72

576

2,880

12,480

37,440

74,880

149,760

Eighteen

108

864

4,320

18,720

56,160

112,320

224,640

A ctivity

D ef

C oQ

D evO ps Econom ics

H ours

R O I

"AG ILE" Te stin g

100

0.1

100 D efects x 70% E ff. x 0.1 Hours

7

8,10 0%

So ftware In sp ectio ns

30

1

30 D efects x 70% E ff. x 1 Hours

21

2,70 0%

"Tra ditiona l" Testing

9

10

9 D efects x 70% E ff. x 10 Hours

63

9 00%

Ma nua l De bug ging

3

100

2.7 D efects x 70% E ff. x 100 Hours

189

3 00%

O peratio ns & Main ten anc e

0.81 1,000 0.81 D efects x 70% E ff. x 1,000 Hours

567

n/a



Traditional testing finds a defect in about 10 hours



Manual code inspections find a defect in 1 hour



Agile testing

finds

a

defect

every

6 minutes

36

Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.

Agile

Testing

—Economics

(37)



Agile testing is

10x better

than

code inspections



Agile testing is

100x better

than

traditional testing



Agile testing is

done earlier

“and”

1,000x more often

37

Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.

Agile

Testing

—Cost of Quality

(38)

Agile Cost & Benefit Analysis



Costs

based on avg.

productivity

and

quality



Productivity

ranged from

4.7

to

5.9

LOC an hour



Costs

were

$588,202

and

benefits

were

$3,930,631

38

Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods: Maximizing ROI with just-in-time processes and documentation.

Ft. Lauderdale, FL: J. Ross Publishing.

d1

= [ln(Benefits



Costs) + (Rate + 0.5



Risk

2

)



Years]



Risk

 

Years,

d2

= d1



Risk

 

Years





5 1

i

(39)

Benefits of Agile Methods



Analysis of 23 agile vs. 7,500 traditional projects



Agile projects are 54% better than traditional ones



Agile has

lower costs

(

61%

) and

fewer defects

(

93%

)

Mah, M. (2008). Measuring agile in the enterprise: Proceedings of the Agile 2008 Conference, Toronto, Canada.

Project Cost in Millions $

0.75 1.50 2.25 3.00 2.8 1.1 Before Agile After Agile

61%

Lower

Cost

Total Staffing 18 11 Before Agile After Agile

39%

Less

Staff

5 10 15 20

Delivery Time in Months

5 10 15 20 18 13.5 Before Agile After Agile

24%

Faster

Cumulative Defects 625 1250 1875 2500 2270 381 Before Agile After Agile

93%

Less

Defects

39





(40)

Agile vs. Traditional Success



Traditional projects succeed at 50% industry avg.



Traditional projects are challenged 20% more often



Agile projects

succeed 3x more

and

fail 3x less often

Standish Group. (2012). Chaos manifesto. Boston, MA: Author.

40

Agile

Traditional

Success

42%

Failed

9%

Challenged

49%

Success

14%

Failed

29%

Challenged

57%



(41)

Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.

Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan.



Most agile testing tools are “free” open source



Build server costs no more than a commodity PC



10x

more

efficient/effective

than

traditional testing

41

Agile

Testing

—CI Statistics

(42)

42



Hewlett-Packard is a major user of CI, CD, & DevOps



400 engineers developed 10 million LOC in 4 years



Major gains in

testing

,

deployment

, &

innovation

Gruver, G., Young, M. & Fulghum, P. (2013). A practical approach to large-scale agile development. Upper Saddle River, NJ: Pearson Education.

T

YPE

M

ETRIC

M

ANUAL

D

EV

O

PS

M

AJOR

G

AINS

C

YCLE

T

IME

I

MPROVEMENTS

Build Time

40 Hours

3 Hours

13 x

No. Builds

1-2 per Day 10-15 per Day

8 x

Feedback

1 per Day

100 per Day

100 x

Regression Testing 240 Hours

24 Hours

10 x

D

EVELOPMENT

C

OST

E

FFORT

D

ISTRIBUTION

Integration

10%

2%

5 x

Planning

20%

5%

4 x

Porting

25%

15%

2 x

Support

25%

5%

5 x

Testing

15%

5%

3 x

Innovation

5%

40%

8 x

Agile

Testing

—CD Statistics

(43)

43

Singleton, A. (2014). Unblock: A guide to the new continuous agile. Needham, MA: Assembla, Inc.



62x Faster

U.S. DoD

IT Project

3,645x Faster

U.S. DoD

IT Project

Agile

Testing

—DevOps Statistics



Goal of continuous delivery is releases vs. build/tests



Market-driven releases creates rapid business value



Assembla went from

2

to

45 monthly releases

w/CD

(44)



Google early adopter of agile methods and Scrum



Google also uses agile testing at enterprise scale



15,000 developers

run

120 million tests

per day

44

Micco, J. (2013). Continuous integration at google scale. Eclipse Con, Boston, MA.

Whittaker, J., Arbon, J., & Carollo, J. (2012). How google tests software. Upper Saddle River, NJ: Pearson Education.



440 billion unique users run 37 trillion searches each year



Single monolithic code tree with mixed language code



Submissions at head – One branch – All from source



20+ code changes/minute – 50% code change/month



5,500+ submissions/day – 120 million tests per day



80,000 builds per day – 20 million builds per year



Auto code inspections – For low defect density



10X programming productivity improvement



$150 million in annual labor savings

(ROI as a result)





Agile

Testing

—Google Statistics

(45)



Amazon adopted agile in 1999 and Scrum in 2004



Using enterprise-scale continuous delivery by 2010



30,000+ developers

deploy over

8,600 releases

a day

45

Atlas, A. (2009). Accidental adoption: The story of scrum at amazon.com. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 135-140.

Jenkins, J. (2011). Velocity culture at amazon.com. Proceedings of the Velocity 2011 Conference, Santa Clara, California, USA.

Elisha, S. (2013). Continuous deployment with amazon web services. Proceedings of the AWS Summit 2013, Sydney, New South Wales, Australia.



Software deployment every 11.6 seconds (as of 2011)



24,828

to

86,320

releases

per Iteration



161,379

to

561,080

releases

per Quarter



645,517

to

2,244,320

releases

per Year



Automatic, split-second roll-forward & backward



75-90% reduction in release-caused outages (0.001%)



Millions of times faster (than traditional methods)



4,357,241 to 15,149,160 per traditional release



Thousands of times faster

(

than manual agility)



161,379 to 561,080 per Scrum/SAFe release



Used agile methods long before U.S. government

(1999)





Agile

Testing

—Amazon Statistics

(46)

Activity Def CoQ DevOps Economics Hours ROI Development Operations 100 0.001 100 Defects x 70% Efficiency x 0.001 Hours 0.070 72,900%

Continuous Delivery 30 0.01 30 Defects x 70% Efficiency x 0.01 Hours 0.210 24,300%

Continuous Integration 9 0.1 9 Defects x 70% Efficiency x 0.1 Hours 0.630 8,100%

Software Inspections 3 1 2.7 Defects x 70% Efficiency x 1 Hours 1.890 2,700%

"Traditional" Testing 0.81 10 0.81 Defects x 70% Efficiency x 10 Hours 5.670 900%

Manual Debugging 0.243 100 0.243 Defects x 70% Efficiency x 100 Hours 17.010 300%

Operations & Maintenance 0.073 1,000 0.0729 Defects x 70% Efficiency x 1,000 Hours 51.030 n/a

46



Agile testing is orders-of-magnitude more efficient



Based on millions of automated tests run in seconds



One-touch

auto-delivery

to

billions

of

global

end-users

Rico, D. F. (2016). Devops cost of quality (CoQ): Phase-based defect removal model. Retrieved May 10, 2016, from http://davidfrico.com



(47)

47



CoQ increases with each life cycle phase



Inspection CoQ is best traditional methods offer



DevOps

yields

orders-of-magnitude

smaller

CoQ

Ambler, S. (2009). Examining the agile cost of change curve. Retrieved March 20, 2012 from http://www.agilemodeling.com



Operations & Maintenance Manual Debugging “Traditional” Testing Software Inspections Continuous Integration Continuous Delivery Development Operations

(48)



Microsoft created software security life cycle in 2002



Waterfall approach tailored for Scrum sprints in 2009



Uses

security req

,

threat modeling

&

security testing

48

Microsoft. (2011). Security development lifecycle: SDL Process Guidance (Version 5.1). Redmond, WA: Author.

Microsoft. (2010). Security development lifecycle: Simplified implementation of the microsoft SDL. Redmond, WA: Author.

Microsoft. (2009). Security development lifecycle: Security development lifecycle for agile development(Version 1.0). Redmond, WA: Author.

Bidstrup, E., & Kowalczyk, E. C. (2005). Security development lifecycle. Changing the software development process to build in security from the start. Security Summit West.



S

EE

D

ETAILED

-

S

ECURITY

L

IFE

C

YCLE

S

TEPS

http://davidfrico.com/agile-security-lifecycle.txt

(49)



Enables enterprises to be flexible but disciplined



Allows enterprises to distribute project work teams



Ensures

distributed

project teams

are

collaborating

49

Agile Tools

“

Across the Life Cycle

”

Project Management Requirements DOORS Requisite Pro SLATE Design Rhapsody

Telelogic System Architect Rational System Architect

Coding Eclipse Visual Studio Sun Studio Testing JUnit NUnit Xunit CPPUnit Gtest Fit Fitnesse Selenium Quality Assurance CheckStyle PMD EMMA Jdepend Cobertura Gcov Configuration Mgt Subversion (SVN)

Concurrent Versions Sys. ClearCase Build Automation Ant NAnt Maven Make Continuous Integ. Cruise Control Hudson BuildBot Collaboration WebEx Skype MeetMe Wimba Wiki MediaWiki TracWiki PhpWiki Documentation NDoc Javadoc Doxygen iText Version One Rally Scrum Works VSTS Agile Team Agile Enterprise Scope Manager Story Studio XP Plan It Iterate XP Tracker Agilo XP CGI XP Web Xplanner Ice Scrum Project Cards Target Process Xtreme Planner Team System Community Enterprise Mingle Hansoft



(50)



There are literally hundreds of agile testing tools



There are tools for building, testing, and deployment



Integration tools

monitor

repositories

and

initiate tests

50

Agile Tools

“

In-Depth Test Automation

”

Smart, J. (2009). Automated deployment with maven and friends: Going the whole nine yards. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.

(51)



Simple example of a DevOps reference architecture



Includes CM, continuous integration, & deployment



Code

automatically

built/tested/deployed

to

users

51

Agile Tools

“

Simple DevOps Automation

”

Morris, B., & Cassatt, C. (2015). Devops for the rest of us. Proceedings of the Agile DC Conference, Washington, DC, USA.

Weeks, D. E. (2014). Devops and continuous delivery reference architectures (volume 1 & 2). Fulton, MD: Sonatype.

(52)

52

Agile Tools

“

Periodic Table of DevOps Automation

”

(53)

53

Holler, R. (2015). Ninth annual state of agile survey: State of agile development. Atlanta, GA: VersionOne.



VersionOne found

94% using agile methods

today



Most are using Scrum

with several key XP practices



Lean-Kanban

is a rising practice with a

31% adoption





Continuous

Integration

●



●

Agile

Testing

—Adoption Statistics

(54)



Agile test use is low in spite of its age, i.e., 15 years



Many do not understand its utter simplicity and power



Failure

to use

agile testing

undermines

project success

54

Kim, D. (2013). The state of scrum: Benchmarks and guidelines. Indianapolis, IN: Scrum Alliance.

Agile

Practices

Retrospectives

Refactoring

Done Definition

Test Tools

Test Driven Dev.

CM Tools

Simplicity

Pair Programming

Technical Debt

Agile Testing

13%

Continuous

Integrations

Weekly

Daily

2-3 Times

Per Day

Never

2-3

Times

Per

Iteration





Agile

Testing

—Usage Statistics

(55)



Agile teams don’t often use TDD, CI, CD & DevOps



Implement independent test teams after Sprints done



Sprint Waterfalling

,

Scrummerfalling

, &

Wagile

result

55

Heusser, M. (2015). 12 years of agile testing: What do we know now. Proceedings of the Agile Gathering, Grand Rapids, Michigan, USA.



Incorrect

• Phased Testing • Separate Teams • Delayed Testing

Correct

• Integrated Testing • Integrated Teams • Continuous Testing

Agile

Testing

—Anti-Patterns

(56)



Agile testing slows down with very large systems



Slow testing slows integration and increases bugs



Agile testing

can

speed back up

with

more attention

56

Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.

M

ICRO

A

DJUSTMENTS

-

Focused Impact Tuning

-M

ACRO

A

DJUSTMENTS

-

Wide Impact Tuning

-Add More CPUs & Memory



Parallelize System Builds



Replace 3rd Party Test Libraries



Reduce or Remove Test Timeouts



Select Different Tests



Refactor Code & Components



Tune Network & Software



Tune Database & Middleware

In-Memory Compilation



Parallelize Test Runs



Pre-Install Test Libraries



Remove Process Randomness



Use Faster Code & Test Tools



Incremental vs. Big Bang Tests



Parallelize Build & Install



Tune & Optimize Build Process



Agile

Testing

—Scaling Practices

(57)



Industry very slow in adopting agile testing model



Cost, difficulty, and territorialism are common issues



Developers

must take

initiative

for

disciplined testing

57

Technical Barriers

Organizational Barriers

Developers don’t want to test

· Infrequently committing code

· Committing broken code

· Failing to immediately fix builds

· Not writing automated tests

· Not ensuring 100% of tests pass

· Not running private builds

· Resorting to traditional testing

Resistance to change

· Fear of investment costs

· Fear of learning new skills

· Test group territorialism

· Organizational policy conflicts

· Overhead of maintaining CI

· Complexity and scaling

· Not developing a quality culture

·





Agile

Testing

—Common Barriers

(58)



Eliminates big-bang integration in the 11th hour



Creates a repeatable and reliable testing process



Evaluates

system-wide changes

throughout project

58

Maeda, M. K. (2009). Agile testing: Early, often, and smart. Arlington, MA: Cutter Consortium.

What’s the Bottom Line?

“

Agile Testing Done Early & Often

”

Agile Testing

Traditional Testing

Dramatically reduces risks

·

Automates manual processes

·

Instant verification & validation

·

High project visibility

·

Greater confidence and morale

·

Incremental business value

·

24x7 deployability to users

·

Highly quality and reliability

Late defect discovery

·

Low quality software

·

Poor project visibility

·

Lack of deployability

·

Late big-bang integration

·

Testing is a bottleneck

·

Poor customer satisfaction

·

Outright project failure

·

(59)

Conclusion



Agile methods

DON’T

mean deliver it now & fix it later



Lightweight, yet disciplined approach to development



Reduced

cost

,

risk

, &

waste

while

improving quality

59

Rico, D. F. (2012). What’s really happening in agile methods: Its principles revisited? Retrieved June 6, 2012, from http://davidfrico.com/agile-principles.pdf

Rico, D. F. (2012). The promises and pitfalls of agile methods. Retrieved February 6, 2013 from, http://davidfrico.com/agile-pros-cons.pdf

Rico, D. F. (2012). How do lean & agile intersect? Retrieved February 6, 2013, from http://davidfrico.com/agile-concept-model-3.pdf

What

How

Result

Flexibility

Use lightweight, yet disciplined processes and artifacts

Low work-in-process

Customer

Involve customers early and often throughout development

Early feedback

Prioritize

Identify highest-priority, value-adding business needs

Focus resources

Descope

Descope complex programs by an order of magnitude

Simplify problem

Decompose

Divide the remaining scope into smaller batches

Manageable pieces

Iterate

Implement pieces one at a time over long periods of time

Diffuse risk

Leanness

Architect and design the system one iteration at a time

JIT waste-free design

Swarm

Implement each component in small cross-functional teams

Knowledge transfer

Collaborate

Use frequent informal communications as often as possible

Efficient data transfer

Test Early

Incrementally test each component as it is developed

Early verification

Test Often

Perform system-level regression testing every few minutes

Early validation

Adapt

Frequently identify optimal process and product solutions

Improve performance





(60)

Dave’s

P

ROFESSIONAL

C

APABILITIES

60

Software

Quality

Mgt.

Technical

Project

Mgt.

Software

Development

Methods

Strategy &

Roadmapping

Systems

Engineering

Cost

Estimating

Acquisition &

Contracting

Organization

Change

Lean

Kanban

Big Data,

Cloud, NoSQL

Workflow

Automation

Metrics,

Models, & SPC

Six

Sigma

BPR, IDEF0,

& DoDAF

DoD 5000,

TRA, & SRA

PSP, TSP, &

Code Reviews

CMMI &

ISO 9001

Innovation

Management

Statistics, CFA,

EFA, & SEM

Research

Methods

Evolutionary

Design

Valuation

— Cost-Benefit Analysis, B/CR, ROI, NPV, BEP, Real Options, etc.

Lean-Agile

— Scrum, SAFe, Continuous Integration & Delivery, DevOps, etc.

STRENGTHS

– Data Mining



Gathering & Reporting Performance Data



Strategic Planning



Executive &

Manage-ment Briefs



Brownbags & Webinars



White Papers



Tiger-Teams



Short-Fuse Tasking



Audits & Reviews



Etc.

●

Data mining

.

Metrics, benchmarks, & performance

.

●

Simplification

.

Refactoring, refinement, & streamlining

.

●

Assessments

.

Audits, reviews, appraisals, & risk analysis

.

●

Coaching

.

Diagnosing, debugging, & restarting stalled projects

.

●

Business cases

.

Cost, benefit, & return-on-investment (ROI) analysis

.

●

Communications

.

Executive summaries, white papers, & lightning talks

.

●

Strategy & tactics

.

Program, project, task, & activity scoping, charters, & plans

.

P

MP

, C

SEP

,

F

CP

, F

CT

A

CP

, C

SM

,

& S

AFE

33+ Y

EARS

IN

IT

I

NDUSTRY

(61)

Books on Agile Testing



Thousands

of textbooks on

agile methods



Include

requirements, design, coding, test, etc.



Continuous Integration

,

Delivery

, &

DevOps

best

61

Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.

Crispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.

Gregory, J., & Crispin, L. (2015). More agile testing: Learning journeys for the whole team. Upper Saddle River, NJ: Pearson Education.

Humble, J., & Farley, D. (2011). Continuous delivery: Reliable software releases through build, test, and deployment automation. Boston, MA: Pearson Education.

(62)

Books on ROI of SW Methods



Guides

to

software methods

for business leaders



Communicates

the

business value

of IT approaches



Rosetta stones

to unlocking

ROI

of

software methods



http://davidfrico.com/agile-book.htm (

Description

)



http://davidfrico.com/roi-book.htm (

Description