Software Testing & Quality Assurance

(1)

Software Testing &

Quality Assurance

(2)

₃

2007 Rank

Software Testing & QA Company

1 Lionbridge

2 UST Global/QA Labs

3 Hewlett Packard

4 EPAM

5 Infosys

6 Trigent

7 LogiGear

8 AztechSoft/Disha

9 IBM Global Testing

10 Wipro

11 Patni

12 Sierra Atlantic

13 AppLabs

14 RTL/Recommened Test Labs

15 Software Quality Systems/SQS Group

16 Accenture

17 ZenQA

18 CalSoftLabs

19 Bleum

(3)

• Innovation and Deployment are the most important attributes influencing Software Testing & QA client satisfaction with their 2007 outsourcing providers. • Strong dissatisfaction is uncommon in this niche ITO

sector, occurring in only 6.7% of Healthcare Industry client types, 11.4% in Software & High Tech

Company clients, and 4.5% of Education clients globally

• Single-vendors offering comprehensive QA services to services software testing clients ranked highest in the overall survey by clients

How the Data Is Collected

Black Book utilizes a three-step process to collect candid performance data. First, Black Book collects a series of direct evaluations covering 26 performance areas on leadership and senior management, which comprise the scored ranking of the “Black Book 50 Best Managed Vendors”.

Second, Black Book collects ballot results on 18

performance areas of operational excellence to rank vendors by outsourced service lines. Third, the gathered data is

Outsourcing, Outsourcing Benchmarkers and Boutique Advisors. These specific survey areas range from four to twenty questions or criteria each.

Statistical Confidence of Black Book Data

Statistical confidence for each performance rating is based upon the number of organizations scoring the outsourcing service. Black Book identifies data confidence by one of several means:

1. Top Ten ranked vendors and advisors must have a minimum of five unique clients represented. Broad categories require a minimum of ten unique client ballots. 2. Vendors and Advisors with over ten unique client votes are eligible for top ten rankings and are assured to have highest confidence and lowest variation. Confidence increases as more organizations report on their outsourcing vendor. Data reported in this form is shown with a 95 percent confidence level (+/- .25, .20 or .15, respectively).

3. Raw numbers include the quantity of completed surveys and the number of unique organizations contributing the data for the survey pool of interest.

Who Participates in the Black Book Ranking

Process?

Over 120,000 individuals are invited to participate annually (including C officers of the Fortune 2000, Inc 500,

institutional members and officers of various professional organizations, subscribers of our media partners and previously validated survey participants). Non-invitation receiving participants must complete a verifiable profile, utilize valid corporate email address and are then included as well. Over 22,000 users were validated in the 2007 ranking process.

The Black Book survey web instrument is open to

respondents and new participants each year from March 1 to May 7 at http://TheBlackBookOfOutsourcing.com

Black Book Methodology

Top Five Honors

1. Lionbridge

2. UST Global (www.ust-global.com) 3. Hewlett Packard

4. EPAM 5. Infosys

(4)

₅

2007 Software Testing & QA Industry Outsourcing

Vendor Rankings and Results

2688 qualified respondents ranked 148 software testing services suppliers offering some 845 contractual

arrangements for testing and or quality assurance service processes as part of the Black Book annual survey conducted Q2 2007 via web survey instrument.

The focus on faster and cheaper in the outsourced software industry is no longer in vogue. Clients are looking for top line contributions, partnerships and extensive capabilities into testing expertise.

Software Risk Management & Analysis

Customized software risk management services encompass project, product and process related risks. Software risk management framework includes:

• Development lifecycle study • Risk identification

• Risk analysis

• Risk mitigation strategy & Implementation

•

Risk monitoring

Full Lifecycle Testing

Software testing at every stage of software development life cycle can prevent defect occurrence and ensure reliable software development for businesses. Next generation software testing provides full life cycle testing services for product and application development life cycles.

Black Box Testing

Black box testing methodology covers the following: • Functionality testing

• GUI testing

• Performance testing • Stress and load testing • Compatibility testing • Installation testing

White Box Testing

White box testing improves the testability of a software product by making testing more effective and efficient. Top vendors provide white box testing services for applications developed using any language. Services include:

• Code coverage • Path coverage • Code analysis

Test Automation

Automation testing drastically reduces testing cycle times. Test automation services encompass white-box as well as

black box testing methodologies. Test automation framework includes:

• Study of development process and solution architecture

• Automation test strategy

• Selection of appropriate testing tools • Test automation implementation

• Customized test automation tools for specific client requirements.

Security Testing

Many enterprise applications and data are vulnerable to external and internal threats. Qualified vendors provide security-testing services for software products, applications and enterprise data. Security testing services include application vulnerability assessment and Risk identification.

(5)

Healthcare

90.5%

Computer Services & High Tech, Software

85.8%

Pharmaceutical/Biotech/Life Sciences

84.6%

Media & Entertainment

80.1%

Consumer Packaged Goods

74.0%

Manufacturing

69.8%

Retail

58.4%

Engineering, Architecture, Construction

53.2%

Transportation & Supply Chain

44.9%

Hospitality & Leisure

37.5%

Banking, Financial & Capital Markets

25.5%

(6)

₇

Comprehensive Outsourced Software Testing & QA Vendor defined as being comprised of three surveyed functions:

1. Independent Analysis, Benchmarking, & Certification

2. Product Testing & Quality Assurance

3. Software Testing & Functional Performance

0.00 – 4.99

5.00 – 6.93

6.94 – 8.00

8.01 – 10.00

• Generally

dissatisfied

• Does not meet

expectations

• Cannot

recommend vendor

• Inconsistently satisfied

• Meets some/few

expectations

• Reserves

recommendation

• Satisfactory

performance

• Meets expectations

• Recommends vendor

• Overwhelming satisfaction

• Exceeds expectations

• Highly recommended

vendor

Operational Excellence of Outsourced Software Testing & QA Firms

Color-coded Stop Light Dashboard Scoring Key

•

Green

(Top 10%).

8.01 +

+ Scores better than 90% of Vendors. Green Coded Vendors received constantly highest client satisfaction

scores.

• Clear (Top 33%)

6.94 to 8.00

Scores better than 67% of Vendors. Well scored vendors that have middle of the pack results.

•

Yellow 5.00 to 6.93

Scores better than 50%of vendors. Cautionary performance scores. Areas of improvement required.

(7)

4

EPAM _8.57

5

Infosys _8.37

6

Trigent _8.01

7

LogiGear _7.90

8

Aztechsoft/Disha 7.81

9

IBM Global Testing _7.72

10

Wipro _7.64

11

Patni 7.63

12

SierraAtlantic _7.50

13

AppLabs _7.48

14

RTL Testlabs 7.35

15

SQS GROUP 7.12

16

Accenture 7.10

17

ZenQA 6.92

18

CalSoftLabs _6.36

19

Bleum 6.27

1 - 6

UPPER MIDDLE

TIER VENDORS

7 - 16

MIDDLE TIER

VENDORS

17 - 57

LOWER MIDDLE

TIER VENDORS

BOTTOM TIER

VENDORS

Ranked

Firms

58 - 90

Ranked

Firms

91 – 114

(8)

₉

Rank Outsourcing Testing Firm 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 MEAN

1 Lionbridge 9.42 9.62 9.17 9.65 9.24 9.00 9.49 9.26 9.04 9.60 9.36 9.78 9.20 9.38 9.40 9.56 9.30 9.55 9.39

2 UST Global/QA Labs 9.16 8.56 9.31 8.73 8.84 9.02 8.96 8.16 7.84 9.08 8.24 8.97 9.21 9.01 9.42 8.97 8.65 8.75 8.83 3 Hewlett Packard 9.04 9.20 8.43 8.78 9.01 6.90 9.57 8.81 9.11 7.39 9.48 8.44 8.99 7.92 8.01 8.11 8.40 9.19 8.60 4 EPAM 9.01 8.05 9.26 8.22 9.30 9.17 9.20 7.75 9.05 7.43 8.91 9.02 8.93 6.55 9.45 8.72 7.79 8.41 8.57 5 Infosys 8.75 8.27 5.98 8.07 9.26 9.13 7.28 8.24 9.18 7.23 9.41 9.59 9.54 6.39 9.10 9.04 7.75 8.43 8.37 6 Trigent 8.89 9.01 7.16 8.95 8.44 5.87 6.92 7.47 5.65 9.03 8.70 7.48 7.65 9.40 7.72 7.49 9.34 9.04 8.01 7 LogiGear 8.17 8.39 8.34 9.17 8.69 5.84 7.98 7.59 6.35 6.57 7.22 9.39 7.62 8.65 7.44 7.25 8.73 8.80 7.90 8 Aztechsoft/Disha 8.71 7.78 8.03 7.17 6.15 8.17 8.97 8.67 8.63 8.67 8.13 8.27 8.47 6.52 6.29 6.58 7.37 8.02 7.81 9 IBM Global Testing 8.65 7.62 8.67 8.02 8.41 5.80 6.32 8.25 8.07 8.20 7.97 8.15 8.33 7.56 5.91 7.63 7.49 7.95 7.72 10 Wipro 8.00 8.73 8.15 8.55 8.13 7.28 7.58 7.07 8.94 8.34 8.69 7.96 7.72 7.20 5.78 5.47 6.84 7.13 7.64 11 Patni 8.60 8.44 9.22 8.24 6.49 7.46 8.30 7.41 7.66 7.47 8.89 7.68 8.00 7.32 5.77 5.92 7.26 7.20 7.63 12 SierraAtlantic 8.38 6.78 8.61 8.91 7.27 6.18 8.27 8.27 7.20 7.92 7.91 7.09 8.03 6.83 6.51 5.93 8.18 6.65 7.50 13 AppLabs 8.52 7.25 7.42 7.77 8.02 6.50 7.91 7.96 8.41 7.10 8.94 6.58 6.95 7.04 6.40 5.90 8.41 7.47 7.48 14 RTL Testlabs 7.51 6.19 5.76 7.65 7.44 7.78 6.38 7.04 8.45 6.12 6.03 6.75 8.72 8.83 8.66 8.41 7.16 7.40 7.35 15 SQS GROUP 8.20 7.37 5.67 7.30 7.45 8.62 7.19 7.88 6.74 8.07 6.73 8.11 8.57 5.97 7.00 5.69 5.90 5.77 7.12 16 Accenture 6.51 5.67 7.18 6.99 6.56 9.04 5.95 7.39 5.49 9.25 7.97 7.38 9.49 5.34 8.78 6.23 6.03 6.56 7.10 17 ZenQA 8.42 7.06 7.99 6.58 7.71 7.60 7.47 6.90 7.03 6.40 4.68 7.22 7.83 5.18 6.48 6.79 6.05 7.22 6.92 18 CalSoftLabs 6.44 5.10 6.14 6.02 5.67 8.55 5.98 7.00 5.60 9.02 6.52 5.79 7.06 5.03 8.35 5.60 5.34 5.19 6.36 19 Bleum 7.22 5.08 6.05 5.66 5.54 8.83 8.51 5.87 5.79 5.98 4.90 6.02 9.00 5.04 7.58 5.18 5.05 5.55 6.27 20 Symbio Group 6.06 5.27 6.38 5.85 4.94 9.06 5.73 6.30 5.46 8.81 6.07 4.18 7.02 4.83 5.83 5.27 5.33 5.28 5.98