MANAGED TESTING SERVICES

MANAGED TESTING

SERVICES

Focus on your core competencies by

test outsourcing

DR DANIEL SIMON Senior Research Manager [email protected]

Daniel Simon studied Computer Science and has been working for SQS since 2005. His main responsibilities include conducting research projects for the Group, evolving and innovating SQS’ service portfolio, and demonstrating its thought leadership. His core competencies relate to outsourcing, technical quality, KPI systems, and quality management. His range of experience covers all sectors of the IT industry, and he is Programme Chair for the iqnite conferences 2012 at Vienna and Geneva.

DR FRANK SIMON Head of SQS Research [email protected]

Frank Simon studied Computer Science and has been with SQS since 2001. He is Head of SQS Research, Head of the BITKOM working stream ‘Software Development Processes and Tools’, and a Member of the German Testing Board. His responsibilities include thinking ahead, innovating SQS’ service portfolio, evaluating hypes and trends, conducting research projects, devel-oping services, and presenting SQS’ research in lectures.

tion has achieved a reasonable level of maturity and is demanding changes towards process and organisational modularisation, standardisation, automation, and focusing on core business com-petencies. If the prerequisites are not yet met, MTS in themselves will be worthwhile to develop an organisation towards the higher maturity level. Since MTS are a specific instantiation of the outsourcing concept, the current status of this market is analysed and an outlook is given on expected future developments for MTS: Managed Testing Services will mainly be delivered by inde-pendent testing companies, blurring the differ-ence to the large full-service providers.

Based on these high expectations, this paper de-scribes the steps towards setting up and operating MTS, focusing on costs and benefits, opportun-ities and risks in the context of these services. Since MTS will often be applied to already existing testing projects, a separate section explains the transition step which is necessary to apply MTS.

As Adam Smith outlined as early as 1776 in his famous ‘An Inquiry into the Nature and Causes of the Wealth of Nations’ (1), progress and a qualitative increase in productivity in society lie in specialisation and industrialism, essen-tially represented by the concept of division of labour. In Smith’s work, the different approaches to manufacture pins were investigated and com-pared, but the same principles apply to the IT in-dustry today and reflect the evolution seen over the past decades.

This whitepaper outlines the concept of Managed Testing Services (MTS), an outsourcing technique for testing-related tasks across one or more pro-jects delivering one or more applications and systems, spanning the life cycle of software and system development and system operation. The paper argues the idea of IT industrialisation as the underlying motivation for using MTS, and shows that MTS are an important step on the indus-trialisation road. It explains the different param-eters that need to be taken into account when de-ciding on IT outsourcing, and gives an overview of important prerequisites that have to be checked before MTS’ benefits can be leveraged.

MTS are a logical consequence of IT industrial-isation in modern IT environments. MTS promise significant savings and quality improvements due to the effects of division of labour and specialisa-tion, as well as exploiting location-independent delivery of well-defined pieces of work. Neverthe-less, MTS are most efficient when an

organisa-This section gives a short overview of the funda-mental basics of so-called Managed Testing Ser vices (MTS). Before a clear definition is provided, it will be shown how both IT and innovations in IT have evolved over the last decades, explaining that Man-aged Testing Services are a manifestation of a more fundamental concept. This overall idea leads to specialisation (i.e. organisations focus on their core competencies), division of labour (i.e. outsourcing in different variations) and (country / cross-cultural) supply chains (i.e. global delivery).

1 MANAGEMENT SUMMARY

2 THE CONCEPT OF MANAGED

TESTING SERVICES

evolve according to the maturity of the busi-ness partners, offering increased predictability of costs as well as flexibility and improved over-all resource utilisation. A more detailed view on these dimensions can be obtained by means of the global delivery cube explained in the previ-ous whitepaper.

Finally, Managed Testing Services also can be directly linked to IT industrialisation since they help customers focus on their core competency, which rarely is in the testing area.

2.2 THE TERM ‘MANAGED TESTING SERVICES’

Managed Testing Services are a combination of managed services and testing activities. Both of these are refined in terms of the innovation YkX[(i[[ <_]kh['_dj^[\ebbem_d]ikXi[Yj_edi X[\eh[WYecfb[j[Z[\_d_j_ede\CWdW][ZJ[ij_d] Services is given.

2.2.1 MANAGED SERVICES

The term ‘managed services’ refers to a busi-ness situation where a customer and a service provider interact in a particular set-up. The customer will consider making use of a service outside his core competencies if he needs it to deliver an important (supporting) process for his core business processes but neither wants to manage the resources required for delivery nor the associated risks. To this end, the entire ser-vice is defined and handed over to the serser-vice provider, making him responsible for the deliv-ery of the respective service in time, in budget, and in quality.

As the IT industry has evolved over time, fun-damental changes to businesses and their sup-porting IT organisations have occurred. Recent years have seen the development from inte-grated full-service units towards specialisation of departments and companies along the vari-ous functions required to run businesses and their supporting IT. Additionally, cost efficiency considerations put high pressure on executives who try to focus on their core competencies and hand over non-core tasks to specialised providers.

2.1 INDUSTRIALISATION: THE ESSENCE BEHIND MANY INNOVATIONS

Growing technical innovations in the IT industry increasingly affect the way in which a service or product is built, delivered, and consumed. The new way IT understands itself is often called industrialisation and is strongly related to the car industry and its history: while the produc-tion-focused sectors (such as the automotive industry) have already undergone significant changes from craftsmanship to an industrialised approach, for IT this is starting right now. There are different dimensions to consider when looking into the details of the IT service industry. The first and most prominent dimension among these would be the vertical range of manufac-ture, which refers to the various engagement styles possible when liaising with businesses. The second dimension addresses the service delivery location and brings attention to the fact that in today’s IT industry various technical means of collaboration across geographical limits are ready for day-to-day use (e.g. delivery of the service through the use of Internet technology, available virtually everywhere in the developed and developing world, or through the use of concepts such as ‘Cloud Computing’). And as a third dimension, the compensation models

Figure ': Global delivery cube with its dimensions BUSINESS PROCESS OUTSOURCING ON SITE ON SHORE NEA RSHORE OF FSHORE APPLICATION OUTSOURCING BUSINESS VALUE-BASED INFRASTRUCTURE OUTSOURCING TRANSACTION-BASED OUTTASKING FIXED PRICE BODY LEASING COST PLUS DIY

with regard to its corresponding specification. The output will be used to remove errors from the test asset and to assess the overall comple-tion of development work. Before the go-live of systems, test results can be used to evaluate the systems’ readiness for use in productive envir-onments and the risks associated with the go-live. Testing nowadays is widely recognised as a mandatory activity in the software development life cycle and should be closely linked to enter-prise risk management since testing identifies defects impeding specific risks if rolled out. In software and systems development, testing activities can start as early as requirements elicitation and can span the entire life cycle down to deployment and maintenance phases of the software and systems. Recent surveys confirm that testing activities are deemed most effective when they are performed by an organi-sational unit that is independent of the develop-ment units. ())

There are international standards for testing (e.g. those provided by the ISTQB), including capability and qualification schemes. Testing is a prime candidate for service outsourcing for customers, not only in order to focus on core competencies but also to exploit the advantage of utilising independent resources. Moreover, many organisations are already used to collabor-ating with testing teams and will have identified and set up their testing capacities in a proper organisational framework. Depending on the maturity of the organisation, tests are executed independently from the development tasks and report to different lines to ensure their inde-pendence. This set-up is widely used and makes it possible to integrate a specialised service pro-vider with reasonable effort.

Managed services can be pinpointed by the fol-lowing dimensions within the framework of the IT service innovation cube:

Engagement styles: Managed services can be applied for infrastructure outsourcing (e.g. managed services for delivering databases), application outsourcing (e.g. managed services for delivering office suites), and even business process outsourcing (see below, Managed Test-ing Services).

Geographical locations: There is no need for a particular geographical favourite since managed services focus on output. However, in many cases nearshore and / or offshore resources are utilised to be able to offer competitive pricing, if legal and regulatory constraints do not hinder it.

Compensation models: In view of the aims of competitive pricing and the realisation of scaling effects, and in order to account for the investment into the set-up of a managed service, the overall volume of the services purchased must be reasonably sized. In most cases, managed services focus on repetitive tasks suggesting a transaction-based compen-sation model. If there is a linkage between the delivered service and the business value, even business value-based pricing might be possible.

2.2.2 TESTING ACTIVITIES

In the IT industry, testing refers to ‘the process consisting of all life cycle activities, both static and dynamic, concerned with planning, prepara-tion and evaluaprepara-tion of software products and re-lated work products to determine that they sat-isfy specified requirements, to demonstrate that they are fit for purpose and to detect defects. (() Typically, the input to testing is the asset under test and the corresponding specification of the asset under test. The output of testing is an analysis of deviations of the asset under test

Output- or transaction-based compensation model: This makes it possible to overcome the time & material-based models and shift enterprise resource risks to the provider. Thus, it simplifies budget planning and generates more budget flexibility.

Leveraging scaling effects on the MTS pro-vider’s side: An MTS provider specialises in testing and he processes many similar tasks around testing which permit setting up the best possible fit-for-purpose infrastructure that is supported by the best-fitting tools inte-grated into an overall smooth toolchain. These scaling effects allow for very quick testing cycles (because usually the provider uses an optimised tool suite for automation) and short response times (because an MTS provider has an established and matured process landscape in place, suggesting dedicated processes for incident management or reporting). High quality of service: For the provider of

MTS, testing is a core competency. He has ac-cess to specialists, best practices, tools and in-frastructures to deliver high-quality services. In addition, many service providers are able to use the advantage of being truly independ-ent of the developmindepend-ent disciplines, as the provider is not integrated into the customer’s organisation (cf. Section 3.3).

Innovation improvement: Using MTS allows customers to refocus on their core competen-cies again. Thus, the services free up custom-er employees’ capacities, improve motivation, and generate new innovation potential. Increased range of the service-oriented

approach: The idea of thinking in service-oriented structures and organisations is not limited to testing but can be expanded to HR, marketing, sales, etc. Using MTS supports this general approach.

2.2.3 THE DEFINITION OF MANAGED TESTING SERVICES

In view of the preceding two subsections, a def-in ition of Managed Testdef-ing Services might read as follows:

Managed Testing Services (MTS) are man-aged services for testing-related tasks across one or more projects delivering one or more applications and systems, spanning the life cycle of software and system development and system operation. The resources for test-ing (staff, testtest-ing infrastructure, system under test) are managed by and under the respon-sibility of the service provider, to support the customer’s business processes. Based on transaction- or business value-based prices, the service provider ensures scalability of the MTS and takes care of the resource manage-ment according to the utilisation scheme re-quired to support the tasks.

Definition 1: Managed Testing Services (MTS)

Managed Testing Services according to this def-inition have the following objectives (cf. (*)):

Complexity reduction: Since MTS mean outsourcing on a high level within the process chain, they reduce the overall process and organisation complexity on the customer’s side by providing a black-box view of it. Risk reduction: MTS providers usually take on

a high degree of risk to deliver MTS. This is underpinned with corresponding Service Level Agreements (SLAs) to shift testing risks to the provider altogether.

Competitive pricing: This is typically realised by a low onshore rate and by moving the geographical focus of the service mainly to nearshore and offshore locations.

3 Automation: For each activity within each process step or even for complete process steps, the possible level of automation has to be considered. There will be activities which require a lot of manual interac-tion, while others might be automated completely.

4 Focus on core competencies: At this level, it can be decided if MTS are possible or not. The fundamental decision whether it makes sense to utilise MTS can be taken if and only if management have a conscious view on what the core competencies of the busi-ness are. This decision can only be taken for identified tasks (modularisation) using standards (e.g. by the ISTQB) and automa-tion to a certain degree.

The decision for or against MTS can only be made if Steps 1 to 3 are successfully completed. Otherwise, it is not clear (i) what testing means, (ii) how testing works, and (iii) to what degree testing can be automated. It is important to point out that a company on its way to Level 4 can already realise significant benefits along the way by improving its industrialisation level. For example, standardisation has values of its own long before MTS can finally be used. And these values may be utilised even without tak-ing the last step of ustak-ing MTS.

MTS volume: Due to scaling effects and ramp-up efforts, the degree to which MTS pay off depends on the size of the testing effort and the number of applications subject to Managed Testing Ser-vices. Typically, the benefits of MTS are realised in three- to five-year engagements and are not achieved in a single project-based liaison be-tween customer and provider. Therefore, the cus-tomer should be prepared to make the strategic investment with the respective support by senior management. For MTS, a customer needs to be in a position to commit to a three- to five-year plan (often known as ‘mid-term planning’). Otherwise, the increased risks for the service provider will

2.3 PREREQUISITES AND BENEFITS

MTS have benefits for both service providers and customers. The benefits for the service providers are transferred to the consumer, e. g. by higher cost efficiency and more competi-tive prices compared to time & material-based projects.

Even though the monetary and non-monetary benefits are widely recognised, it should be noted that the exploitation of these benefits comes with prerequisites. If these prerequi-sites are not met on either the customer’s or the service provider’s side, the risk of false promises and failure to achieve the objectives is high. However, a well-defined transition pro-cess like the one presented in Section 4 will identify those gaps as early as possible and will bridge them efficiently.

2.3.1 PREREQUISITES

There are three different types of prerequisites that should be fulfilled before leveraging MTS’ benefits:

Industrialised process chain: First of all, the organisation and processes on the customer’s side must be mature enough to support the division of labour across organisational units. This typical industrialisation state has at least four prerequisites:

1 Modularisation: The overall business process chain must be refined to a level where a detailed process landscape can be identified. This process landscape defines activities, responsibilities, dependencies, and results. 2 Standardisation: Based on the

modularisa-tion, the process landscape as well as the interfaces must utilise standards. Applying those standards guarantees a specific level of quality and makes establishment easier due to an abundance of existing literature and knowledge.

tion, the best possible organisational set-up for running independent tests has been chosen. Both of these aspects will have a positive impact on the overall effectiveness of the testing process, and will ultimately result in a higher quality of the applications and systems being tested (cf. Section 3.3).

In many cases, the volume of the MTS is high enough to justify investments to leverage cross-project and system synergies in terms of test au-tomation and systematic regression testing. Not only does this increase the effectiveness of the testing service but it also reduces response times from the testing functions to the development or-ganisation.

If the MTS provider can make use of offshore testing in different time zones and the delivery processes are set up properly, a ‘follow-the-sun approach’ of MTS delivery can be achieved which will dramatically reduce the turnaround times of testing. This means that the development and the testing capabilities in a project can be used in a 24-hour schedule by following the time zones. On the other hand, this approach requires mature processes and a significant amount of trust be-tween the different parties to be successful. From the perspective of the customer, a signifi-cant risk and management effort concerning the resources is delegated to the service providers. Since the latter will have different customers with different schedules, the scalability of resources in the project life cycle is no longer an issue for the customer and will be managed by the service pro-vider. Therefore, the customer’s internal resour-ces can be released from non-business-related tasks and can focus on his business core activi-ties. He will still need to contribute and provide in-put, but this can happen in a more systematically structured way and thus will result in both higher quality and less effort.

have to be compensated for by higher prices. Openness for change: Setting up MTS in an

organisation implies the shift of labour from internal resources (including body leasing or outtasking) to external resources having their own responsibilities. In pure MTS, the external resources are even no longer visible to the customer’s staff. Thus, for most customers changes will occur that need to be managed appropriately, and management must be fully aware of the consequences for internal resources and address concerns proactively. Ideally, the customer will already have experi-ence in the field of supplier management and can build upon this experience to collaborate with the service provider.

2.3.2 BENEFITS FOR CUSTOMERS

Key benefits for service consumers can be real-ised over the run-time of the MTS agreement, most prominently the cost savings resulting from transparent pricing models such as output-based pricing, as well as delegated procurement of test infrastructure and tooling. If set up properly, cus-tomers will see an increase in transparency of cost with regard to their application life cycles. Note that this does not necessarily imply lower pricing – but prices are made explicit rather than being hid-den in various budgets. One additional benefit is provided by cost flattening: usually, the ramp-up phase needs more effort than the exe cution of MTS. However, this peak effort is not directly in-voiced to the customer, who instead pays a flat fee over the whole life cycle. This is a key enabler for management as an engineering discipline. As MTS will be delivered by service providers having expertise in the field of quality assurance and testing, it can be assumed that these service providers have highly skilled resources and mod-ern technical environments. If the MTS providers are independent of the development

organisa-on staffing needs. The significant sizing of MTS allows the service provider to set up core teams that facilitate an internal handover in case of staff fluctuations or peak workload. He also has the op-tion to build up internal core expert teams which will be cross-pollinated from testing expertise and customer / domain expertise. At the same time, resource demands from different projects (even across different customers) can be managed on a more flexible and scalable basis.

2.3.3 BENEFITS FOR PROVIDERS

Providing MTS to a customer has a number of advantages over time & material projects for the service providers. Typically, MTS are aligned with an overall strategic sourcing decision of the service customer, and a strategic partnership is established for delivering MTS. Strategic partner-ship decisions usually cover a timeline of three to five years’ commitment and accordingly give the service provider a security of revenue for his own planning, along with an enhanced forecast

3.1 OUTSOURCING AND CLOUD COMPUTING

Outsourcing as the overall approach behind MTS is already a well-known and established concept. However, in the IT industry, maximum coverage has not yet been achieved. Typical trends in out-sourcing and how they have evolved over the last few years as a percentage of total IT budgets are given in Figure (, based on a study published by Computer Economics. (+)

3 MARKET – CURRENT STATUS AND

OUTLOOK

Figure (: Outsourcing ratios as a percentage of total IT budgets over the last three years

These numbers clearly demonstrate the overall trend of IT industrialisation: a growing number of companies use third parties to deliver IT services. And this trend is set to increase over the next few years. A study conducted by market research firm PAC expects an outsourcing ratio of about 25 % by 2020 – this means that one out of four euros in the IT market will be spent on outsour-cing. (,)

Some studies refer 40 % to 50 % of the cost of software development to testing (.), others are measuring a ratio of 1:1 for development to test effort (/). In a recent survey conducted on the occasion of the iqnite 2011 conference, 65 % of the participants reported spending between 20 % and 40 % of their overall project budget on qual-ity management and testing activities. ()) However, most of the testing budget is spent on system integrators – which contradicts the best-practice approach of separating product develop-ment from system testing. In a PAC study, 78 % of the overall quality assurance budget was spent on system integrators and only 22 % was spent on dedicated and independent testers. ('&) This is expected to change significantly over the next few years, as currently 57 % of companies still do testing on the DIY level and will need to mature in that respect.

3.3 FULL-SERVICE PROVIDERS VS. TEST SERVICE SPECIALISTS

In general, Managed Testing Services can be pro-vided by two different types of providers:

Test service specialists: Specialist providers typically offer services in the subject mat-ter ‘testing’ only. They do not offer forward development (implementation services), they do not design software architectures, nor do they have any particular interest in (or benefit from) any proprietary solution compromising their independent approach. The advantage of those players is that they are independent of all systems under test. According to most known testing literature, being independent is one of the most important prerequisites for ef-ficient testing. Usually, test service specialists fulfil this requirement by being an independent test organisation (see Definition 3).

Today, outsourcing is strongly associated with the usage of cloud services and cloud computing in general. For the following, we will use Gartner’s definition of ‘Cloud Computing’:

Public cloud computing [is a] style of com-puting in which scalable and elastic IT-en-abled capabilities are delivered as a service to external customers using Internet technolo-gies. Private cloud computing is defined as a style of computing in which scalable and elastic IT-enabled capabilities are delivered as a service to internal customers using In-ternet technologies.

Definition 2: Cloud computing (-)

Viewed from the outsourcing perspective, cloud services are a particular type of outsourcing with some additional constraints. Nevertheless, cloud computing’s part of the outsourcing market will increase from less than 20 % today to 70 % in 2020. (,) A direct comparison between the def-inition of MTS and the definition of cloud services reveals a high similarity between MTS and cloud computing because MTS are scalable and elastic as well, most of the delivery of MTS is IT-enabled, and the results can be provided using Internet technologies.

This similarity suggests that MTS will have a simi-lar rate of growth and can be delivered by means of cloud computing.

3.2 TESTING SERVICES

The general overview of the IT market provided in the previous section can be refined to focus on MTS. Both academic research and industry experi-ence report a range of figures on how much test effort is necessary and reasonable for a project. direct comparison between the

def-others are In a recent survey conducted on the

The main advantages of using a full-service pro-vider are:

Company size: A broader portfolio of services and tools allows for a bigger market enabling a bigger company size. Typically, full-service providers are known globally and have a large pool of employees at their disposal.

All-inclusive single sourcing: Using a full-service provider simplifies both the process of selecting third parties and the negotiation step. Having a contract with a full-service pro-vider means getting everything through one contract, with one key-account person – it is a one-stop shop.

Technical knowledge: Introducing an independ-ent test organisation to a new IT system costs more time and money than using one of the service provider’s developers. The full-service provider has deep knowledge of the technolo-gies he develops and provides, which enables him to start working without having a clear knowledge-transfer step.

Both independent test organisations and full-ser-vice providers will try to move into the Managed Testing Services market (see Section 2.3.3). How-ever, the market will be further segmented: the more mature an organisation is (e.g. in terms of industrialisation), the more important attributes like provider exchangeability, dependency reduc-tion, and output-based pricing are going to be. These considerations assume an explicit testing awareness on the customer’s side. If it is given, the application of MTS is a natural step, i.e. MTS will successfully be provided by test service spe-cialists.

Figure ) shows the total revenue generated by in-dependent test organisations in 2010.

An independent test organization is an or-ganization, person, or company that tests products, materials, software, etc. according to agreed requirements. […] They are inde-pendent because they are not affiliated with the producer nor the user of the item being tested: no commercial bias is present.

Definition 3: Independent test organisation ('')

The key advantages of using an independent test organisation are the following:

· Independence: Independent people are more likely to question everything, provide an objective second opinion, identify implicit requirements, and objectively report any find-ings. Different lines of reporting ensure that no conflicts of interest will interfere.

· Scaling effects: Because independent test or-ganisations are usually pure-play testers, they test many different projects at the same time, which allows them to set up highly sophisti-cated toolchains and utilise a large number of synergies between different projects.

· Resource capabilities: The skills provided by a test service specialist are clearly focused on one single topic, i.e. testing. This guarantees a high, coherent skill level for all employees and simplifies the exchange of resources. Full-service providers: These providers

usu-ally offer any services relating to IT systems, starting from collecting requirements, design-ing system architectures, implementdesign-ing the entire system, testing the system, and finally deploying the system into productive use. Also administration of the systems and testing services, including Managed Testing Services, form part of the comprehensive service port-folio. In many cases the full-service providers even market their own tool platforms, comple-menting and mixing their service ideas with software license vending.

testing approach to MTS may prove difficult. In other words, MTS are likely to remain a side prod-uct for full-service providers, and consequently only about 5 % of all testing activities provided by them will be delivered as MTS by 2015. Using the revenue of the full-service providers listed in the NelsonHall Report, this amounts to an overall business size of up to _{€ 150 million of revenue for} MTS (5 % of the total market size as listed in ('()). Consequently, the independent test organisations will reach the same market share for MTS as the full-service providers.

Currently, conservative estimates anticipate about 20 % to 25 % of all testing activities provided by test service specialists to be realised through MTS by 2015. For the independent test organisations listed, this amounts to an overall business size of about _{€ 121 million of revenue for MTS (25 % of} the total market size as listed above).

On the other hand, full-service providers usually blend testing activities and package them into an all-inclusive bundle. If the client lacks awareness of what really is involved in testing, testing effort and testing outputs, motivating him to change the INDEPENDENT TEST ORGANISATIONS

163 70 40 28 20 16 14 13 13 12 11 10 9 9 8 7 7 7 6 6 5 5 5 0 20 40 60 160 SQS Applabs* Tescom DNV-IT Tesnet Plan IT Thinksoft Imbus FHG / IESE Amsphere MTP Acial ps-testware Quality House Maverick QA Infotech RTTS Genilogix Squerist Experior Immune Assurity KJR

Figure ) : Revenue of independent test organisations ('()

* In 2 011, Applab s w as t a k en o v er b y C S C C omp an y. €m

The Pre-Engagement step covers the strategic activities a client needs to perform in order to ap-ply MTS. The MTS Framework Design step covers the tactical activities within each client’s project to prepare MTS’ application. And the MTS Opera-tion step leverages the benefits associated with MTS. Each of these steps is refined in the follow-ing subsections.

4.1.1 PRE-ENGAGEMENT PHASE

At the very beginning of the transition to MTS, the customer needs to define and commit to the vision of industrialising his IT environment (see Figure +). On the one hand, management need to identify the strategic services and core competencies they want to operate on their own behalf, and on the other hand, they need to specify the supporting non-core services to be outsourced.

The vision to industrialise IT and make use of ser-vice providers has to be supported by conscious planning and budgeting decisions. A success-ful migration to an outsourcing model demands proper preparation of the underlying business case. Before MTS can be deployed into the or-ganisation, the strategic sourcing decision must

be taken and at least mid-term planning perspec-tives have to account for handing over a com-plete process to a service provider. The strategic decision is a sine qua non because of the sizing of the engagement. Typically, using a managed service requires the investment of significant ef-forts into organisational maturity, and the corre-sponding changes have to be communicated and managed appropriately.

The final step of the Pre-Engagement phase is to set up the relationship to potential service pro-viders and selecting the best supplier. For MTS, criteria for supplier selection start with match-ing the three dimensions from the IT innovation cube (engagement style, geographical location, and compensation model, cf. Section 2.1) to the customer’s needs. Additional parameters to be considered are the following:

Level of testing expertise

Independence from the development organisation

Domain knowledge Size of the provider Commercial background

Track record and reputation of the service provider

4.1 TRANSITION OVERVIEW

The MTS process is structured into three phases – Pre-Engagement, MTS Framework Design, and MTS Operation – and laid out in Figure *.

Figure *: MTS process overview

1. PRE-ENGAGEMENT 2. MTS FRAMEWORK DESIGN

3. MTS OPERATION

4 MANAGED TESTING SERVICES SET-UP

AND OPERATION

Heterogeneous views on core business com-petencies and unclear strategic considerations with regard to sourcing decisions

Depending on the results from the maturity as-sessment, different initiatives to industrialise and professionalise IT will be launched as business processes are set up, streamlined, and optimised. Preparing the organisation for MTS increases the necessity to outline the strategic vision of future IT set-ups, and makes the corporate strategy vis-ible to all employees.

Accordingly, the typical set-up for an IT improve-ment towards industrialisation consists of the four steps mentioned in Section 2.3.1:

1 Modularise the entire value chain within the organisation.

2 Standardise the process landscape, the pro-cesses themselves, and their interfaces. 3 Automate single processes, process

interac-tions, or process chains, wherever possible. 4 Focus on core competencies and delegate

per-ipheral services to external service providers as managed services.

The benefits of preparing for MTS are tangible for customers when MTS are being used as a driver for maturing the organisation. As a side effect of the MTS Framework Design phase, customers need to reconsider their internal processes and organisational set-up. The maturity assessment of processes and organisation can be leveraged to foster clarity and transparency of core

com-4.1.2 MTS FRAMEWORK DESIGN PHASE

One of the greatest challenges when setting up MTS lies in the degree of IT industrialisation on the consumer’s side. Clearly, without an explicit organisational set-up and mature processes, the realisation of benefits from division of labour and the consumption of specific processes / services from the outside are rarely possible. Therefore, the first step in bringing MTS to life is the assessment of the customer’s current organ-isation and its processes to evaluate their fitness for purpose (see Figure ,). The assessment typic-ally unveils a number of necessary improvements to close the gaps and achieve the adequate level of maturity. However, these improvements do not only affect MTS readiness but essentially generate added value even without using MTS.

In our experience, customers aiming at the use of MTS (or other managed services) often over-estimate their readiness for the consumption of services. Typical unrecognised shortcomings are the following:

Insufficiently structured and defined pro-cesses, tasks, and supporting IT landscape as well as undefined or unclear ownership of the process or IT components

Insufficient deployment of process and IT stand-ards (e.g. non-existent or incomplete process landscape, high number of variations in business processes, unclear and undocumented inter-faces between processes and departments) High proportion of manually executed

pro-cesses and technological and organisational ruptures in process chains as well as deficien-cies in process interactions

Figure +: Pre-Engagement activities

VISION TO INDUSTRIALISE IT STRATEGIC PLANNING CONSIDERATIONS (MID-TERM PLANNING) PROVIDER PRESELECTION

typic-4.1.3 MTS OPERATION PHASE

After having set up the general MTS framework for collaboration between the customer and ser-vice provider, the testing activities for the indi-vidual applications have to be transferred from the customer’s responsibility to the aegis of the MTS provider. For greenfield projects, this seems a straightforward task whereas for existing pro-jects there is potentially more substantial change involved. In any case, the readiness for transfer is ensured in the first step, where potential gaps are identified and closed accordingly (see Fig-ure -). Secondly, the knowledge transfer (both from the development organisation to the MTS provider, and from the test organisation to the MTS provider) is initiated, and the first iteration of delivery is accomplished jointly. This collabora-tive delivery is a one-off undertaking designed (i) to ensure a successful transfer of the testing activities to the third step (Service Operation) by passing on comprehensive knowledge to the MTS provider, and (ii) to sort out any issues that might arise during stand-alone delivery to the best pos-sible extent.

petencies and non-core supporting functions. It is worth mentioning that through the course of process improvement, the relationship with the service provider can be established and the trans-formation can be used to build mutual trust. Finally, when readiness has been accomplished, the customer needs to design the framework agreements with the service provider of choice. Utilising Managed Testing Services requires meth-ods to monitor and control service delivery. The challenge lies in a proper definition of the SLA: due to the black-box view of MTS, it should focus on the compensation model rather than on inter-nal MTS details. In order to link the KPIs to the compensation model and evaluate them accord-ing to a reasonable schedule, a sound framework needs to be defined. From the very beginning, the definition of KPIs should take into account the evolution of service level objectives over time to establish appropriate productivity goals. For moni-toring and controlling purposes, these KPIs are utilised for an effective governance and commu-nication structure and process, so as to be able to react when the control limits are reached.

Figure -: MTS Operation processes

CONTINUOUS IMPROVEMENT VERIFICATION OF TRANSFER READINESS PER APPLICATION KNOWLEDGE TRANSFER & COLLABORATIVE DELIVERY SERVICE OPERATION SERVICE COMPLETION

Figure ,: MTS Framework Design activities

ASSESSMENT OF CURRENT ORGANISATION AND PROCESSES PROCESS AND RESOURCE TUNING FRAMEWORK AGREEMENT WITH PROVIDER

laborative Delivery step, it helps determine and fine-tune the agreed KPIs of the framework agree-ments and can be used to adjust the SLA. While Service Operation is repeatedly executed, Con-tinuous Improvement ensures that the KPIs are properly monitored and adequate action is taken to improve and control the delivery. The same KPIs used to report back on service performance should also be applied to manage and govern the MTS Operation phase. Since the benefits based on improvements affect both consumer and MTS provider, the sharing of the benefits needs to be clearly defined in the SLA.

Out of the four MTS phases, the MTS Operation phase is the one covering the longest time span – it implies a shift from a project governance struc-ture towards a programme governance strucstruc-ture. A stereotypical organisational set-up is shown in Figure ..

Subsequently, the Service Operation step may be executed as often as necessary over the lifetime of the application, until this lifetime comes to a close and both the application and its MTS are decommissioned in the Service Completion step. Since the Service Completion step is the reverse of the transition step, planning for this step helps limit dependency on a specific MTS provider. The Service Completion step takes care of gathering all information (including internal details like test scripts, test data, or test procedures) necessary to deliver MTS for a particular project. This allows the customer to hand over the collateral informa-tion to any other MTS provider, or to insource the testing activities at a later point in time, ensuring a shorter introduction time.

The sequence of four processes is overarched by the ‘Continuous Improvement’ of the MTS Opera-tion phase. During the Knowledge Transfer &

Col-Figure .: Governance structures for the MTS Operation phase

PROJECT GOVERNANCE MANAGED SERVICE GOVERNANCE

EXECUTIVE STEERING COMMITTEE

SERVICE DELIVERY COMMITTEE

INNOVATION & TECHNOLOGY COMMITTEE

Test Factory MS Delivery

Management Applications

Testing

Services Test Team

Testing

Services Test Team

Testing

Services Test Team

Functional Sub-Projects Testing Services Technical Sub-Projects Quality Assurance Project Management

proven governance structure consists of the fol-lowing decision bodies in a strategic and an op-erational layer:

STRATEGIC LAYER

Executive Steering Committee (ESC) – Meet-ings are held quarterly (or as described in the framework agreement) as senior management meetings to monitor overall MTS performance, to govern MTS from a long-term perspective, and to de-escalate issues arising from the operational layer.

Innovation & Technology Committee (ITC) – The ITC meets quarterly to share the latest industry, innovation and technical trends, in order to decide on major innovation projects and jointly agree on business cases and fund-ing, if required. Its members propose changes to the agreements to the ESC and also consult the ESC on strategic decision-making.

OPERATIONAL LAYER

Service Delivery Committee (SDC) – The SDC performs weekly reviews of the current perform - ance and progress, focusing on the domain, project and service levels. It decides on prior-ities, resolves escalation issues as best pos-sible, and elaborates on how to mitigate risk and issues within the scope of the services’ operation. Issues outside the SDC’s responsi-bility are referred to the ESC.

Operational Status Meetings – These meetings are scheduled weekly and aligned with core testing processes in order to review progress, focusing on the project or application level, and to decide on priorities and work on risks and issues, seeking for resolution wherever possible or referring them to the SDC. Project Governance rests entirely with the Project

Management (which then reports to the executive level). The Project Management function – sup-ported by the Quality Assurance function – per-forms Testing Services as per project by the same hierarchical decomposition as the Functional and Technical Sub-Projects. In the MTS setting, the Testing Services liaise with the Quality Assurance function and are managed as per application by suppliers interfacing with the respective project teams. The reporting lines to the executive level are no longer purely bottom-up but complement the internal reports with an independent view on projects and applications from the supplier’s per-spective. This typically gives customer manage-ment deeper insight into the project status and increases trust in the reports.

As a general principle, the MTS Operation phase must be supported by an open dialogue between customer and service provider relating to the provision of the services throughout the term. In order to keep up with technological changes over the run-time of the overall agreement, details of the agreement may need to be amended, whether as a result of market trends, new technology, evolution of new services, or otherwise, and must be kept under constant review, control, and sub-ject to proactive search for improvements in the MTS Operation phase. To facilitate the dialogue, both partner organisations typically appoint a Re-lationship Manager at senior management level to cover all matters concerning the agreements. The service provider appoints a Service Delivery Manager as the point of contact for operational matters affecting the services (e.g. adding new applications to the portfolio; customer projects ordering services).

Having defined the roles, a corresponding formal and structured channel of escalation and com-munication has to be set up and maintained. The

4.2.1 INEFFECTIVE TESTING

If testing is ineffective, the uncertainty about the system to be rolled out increases because nobody knows about possible hidden errors in produc-tion and potential impacts. Since risks are based on uncertainties, the overall risk of an IT system increases as well. This does not necessarily im-ply that the system fails in production but that it lacks the necessary transparency and reliable insights about production readiness. Therefore, ineffective MTS for a business-critical application might induce additional risk.

From an MTS perspective as described above, the following countermeasures are to be taken:

Within the transition step, many prerequisites for MTS are verified. The transition step will identify most of the detailed hurdles for ef-fective testing and eliminate the respective reasons.

Before MTS enter the operational phase, the collaborative delivery is an explicit mitigation activity to ensure in a dry-run that testing can be implemented as planned.

The MTS process incorporates the definition of an overall control framework underpinned by supporting KPIs to make the status of the testing process transparent to all stakehold-ers. Therefore, uncertainties about the test progress should be minimised.

4.2 MTS RISK MANAGEMENT

As with all other business activities, there are risks associated with MTS. For risk, we use the following definition, based on the concept of un-certainty:

Uncertainty is the lack of complete certainty, that is, the existence of more than one possi-bility. […] Risk is a state of uncertainty where some of the possibilities involve a loss, catas-trophe, or other undesirable outcome.

Definition 4: Uncertainty and risk ('))

So each risk is an uncertainty but not vice versa. Uncertainties can have both positive or negative impacts, risks only have negative impacts on busi-ness. Please note that this understanding of risk is more focused than the one used in ISO 31.000 since there even positive impacts are classified as risk.

From a risk management perspective, the focus is on negative risks because they require mitigation actions to deal with their impacts.

For a customer, MTS involve two different types of risks:

Ineffective Testing: Testing does not work properly, i.e. the SLAs for the MTS or the stra-tegic objectives to do MTS are not fulfilled. Unfulfilled Collaterals: Testing works but other

collaterals are uncertain. This uncertainty can cover the economic future of the service pro-vider or might touch the cultural acceptance of MTS within an organisation.

Both risks are refined in the following subsec-tions.

Communication risks: In general, MTS need more mature communication channels (no longer ‘gossiping’) when communicating from customer to testing. In addition, a more mature defect management is necessary since the organisational boundaries discourage on-the-floor communication. And last but not least, cultural and language issues are a major source of misunderstanding, in particular if cross-national and offshore locations are involved. MTS might generate risks in this field, but the industrialisation steps, that are checked on the customer’s side in advance, as well as the systematic transition step minimise them.

Dependency risks: Outsourcing is often associ-ated with outsourcing responsibilities, i.e. test-ing capabilities for MTS. The less knowledge is retained on the customer’s side the more he is dependent on MTS providers. Typ ical examples of a high dependency are price adjustments without any reason, disburdening details of the negotiations, or low quality. In all cases, MTS propose the well-defined Service Completion step, ensuring that all custom-ers can step back. Due to the detailed entry criteria (industrialisation), the customer has a very clear understanding of the dependency, possible impact, and alternatives for delivery.

4.2.2 UNFULFILLED COLLATERALS

Outsourcing a well-defined service might stipu-late uncertainty in some colstipu-lateral areas. Most of them are not solely related to MTS but to all major changes in the context of service delivery. In a study about risks of outsourcing (not limited to MTS), a set of most common risks was identi-fied by survey respondents. ('*) Most of them are valid for MTS as well:

Pricing risks: In many cases, using MTS should have a positive effect on costs. However, the specific costs of MTS are not ultimately known before the first Service Operation step. The MTS framework provides general guidance but only the Collaborative Delivery will determine the final price. From a customer perspec-tive, a mitigation action would be to select an experienced MTS provider whose first effort estimation does not need much adjustment. If there are price adjustments while the service is delivered, the Service Completion step en-sures that it is possible to change the service provider to a cheaper one.

Planning risks: If some other constraints from MTS are not met (e.g. missed deadlines, un-clear planning), this might generate additional risks as well. The MTS process guarantees to identify deviations as early as possible. In most cases, any deviations are followed by contrac-tual penalties compensating negative effects by planning risks.

The market for MTS will increase dramatically over the next few years. Even if both, full-service providers and test service specialists, will try to do business in that area, the test service special-ists will dominate the market in a few years’ time due to many advantages in their MTS delivery. Their process to establish MTS on the consumer’s side reflects a well-defined current-state analysis as well as a corresponding transition step for MTS readiness. Based on a continuous improvement loop, MTS execution itself can be managed ef-fectively. A possible closing step at the end can reflect an application’s end as well as the demand to in-house all activities again.

Today, testing is established as an important dis-cipline for risk management. For maximum ef-ficiency, the testing activities, the methods, the practices, tools and skills should only be of inter-est to those who execute tinter-esting. From a business perspective, testing should be seen as a black box focusing testing’s output. So there is a great op-portunity to outsource testing activities at the rate of output-based pricing: Managed Testing Services.

Applying MTS is the next step of IT industrialisa-tion and needs a mature process landscape on both the consumer’s side and the producer’s side. Even if there is no need to apply MTS as a last step on the industrialisation roadmap, the road itself generates a considerable amount of added value for improving testing efficiency and effect-iveness.

1 Adam Smith. An Inquiry into the Nature and Causes of the Wealth of Na-tions. Oxford University Press, 2008.

* Dr Wolfgang Balze, Dr Wolfgang Rebel and Peter Schuck. Outsourcing und arbeitsrechtliche Restrukturierung von Unternehmen. Verlagsgruppe Hüthig Jehle Rehm, 2007.

+ Computer Economics. IT Outsourcing Statistics 2010 / 2011: Outsourcing and Offshoring Trends, Cost/Service Level Experiences, and Analysis for 11 Outsourced IT Functions. California, Irvine: Computer Economics, 2010. , Karsten Leclerque. Outsourcing im Jahr 2020. www.cio.de. [Online]

09/09/2010. [Cited: 19/07/2011.] http://www.cio.de/knowledgecenter/out-sourcing/2247905/index.html.

- Daryl C. Plummer, David Mitchell Smith, Thomas J. Bittman, David W. Cear -ley, David J. Cappuccio, Donna Scott, Rakesh Kumar and Bruce Robertson. Five Refining Attributes of Public and Private Cloud Computing. Gartner, May 2009.

. Praveen Ranjan Srivastava. Estimation of Software Testing Effor t: An intel-ligent approach. Birla Institute of Technology and Science, Pilani, Rajasthan, India: Computer Science and Information System Group, 2009.

/ Michael A. Cusamano and Richard W. Selby. Microsoft Secrets. New York: Simon and Schuster, 1995.

'& PAC. Compliance and Software Testing. Compliance Magazine. [Online] 04/04/2007. [Cited: 19/07/2011.] http://www.compliancemagazin.de/markt/ studien/pac040407.html.

'' Wikipedia. Independent test organization. Wikipedia, the free encyclopedia. [Online] 27/07/2011. [Cited: 27/07/2011.] http://en.wikipedia.org/wiki/Independ-ent_test_organization.

2 International Software Testing Qualifications Board, Glossary Working Party. Standard Glossary of Terms Used in Software Testing. Homepage: ISTQB, December 2007.

) iqnite conference. iqnite interaktiv SMS voting. iqnite germany program. [Online] 01/07/2011. [Cited: 28/07/2011.] http://www.iqnite-conferences.com/ Trial%20Wiki/iqnite_interaktiv_sms-voting.aspx.

Dr Wolfgang Balze, Dr Wolfgang Rebel and Peter Schuck. Outsourcing und

Computer Economics. IT Outsourcing Statistics 2010 / 2011: Outsourcing

Karsten Leclerque. Outsourcing im Jahr 2020. www.cio.de. [Online]

Wikipedia. Independent test organization. Wikipedia, the free encyclopedia.

intel-'( NelsonHall Report. Independent Testing Providers vs. System Integrators. Cologne: SQS, April 2010.

'3 Douglas Hubbard. The Failure of Risk Management: Why it’s broken and how to fix it. John Wiley & Sons, 2009.

'* Diamond Management & Technology Consultants. 2006 Global IT Outsour-cing Study. Chicago: Suite 3000 John Hancock Center, 2006.

Cologne: SQS, April 2010.

to fix it. John Wiley & Sons, 2009.

51149 Cologne / Germany

www.sqs.com