Software Development Risk

Software Development Risk

Assessment: A

multi-dimensional, Quantitative &

Continuous Approach

Theme: STRATEGIC & INNOVATIVE PRACTICES PORTFOLIO,

PROGRAMS & PROJECT (PPP) MANAGEMENT FOR

REDEFINING INDIA

6/18/2015

PMI India National Conference 2015

Mona Mahatha

Abstract

Software Risk management has emerged as one of most critical areas of project management and indicator of project outcome. Formal risk analysis and management is a key focus area for project managers these days as it is intended to yield significant benefits and determine project success. Risks are also inevitably linked to cost. In large and complex projects, risks linked additional costs assume higher magnitude and are seen to have considerable impact on project budgets.

In actual practice of risk management, the focus and effort is more on identifying comprehensive set of risk factors, creating mitigation and/or contingency plans to manage these risks, and there is considerably lesser understanding and practice of evaluating costs linked to risks and monitoring these costs over the risk management cycle. This paper intends to fulfill this gap by providing project managers with a re-usable, intuitive and easy to use framework to enable them to list software project risk factors, to categorize risks as per their dominant risk outcomes and evaluate associated cost impacts quantitatively and continuously over the project life-cycle.

Monitoring and recording absolute impact of risks will also help create an organizational baseline and valuable database to assess risk linked costs and utilizing it for more realistic effort estimations for later projects or release cycles.

Keywords

Contents

1

Introduction

... 4

2

Risks & Risk Management: Background and related studies

... 5

3

Risk Impact Assessment

... 5

3.1

Challenges in Risk Assessment and Recommendations in Proposed framework

... 7

3.1.1

Determining combined risk exposure of project in view of overlapping or linked risk

events.

7

3.1.2

Distinction between current and incurred risk exposure in projects

... 10

3.1.3

Integrating multiple impact dimensions of risks

... 11

4

Proposed Risk Assessment Framework

... 12

5

Summary & Conclusion

... 13

1 Introduction

Software development projects are endeavors which involve interaction of several human and non-human factors leading to complex inter-dependencies and making them susceptible to failures. It is also observed that despite growing levels of project management maturity and evolution of varied project management models, a software project’s predictability remains elusive and impacted by certain events that can change the course of project execution and its outcome. As a result several software projects suffer multiple points of failures such as schedule slippages, cost overruns or failures to meet their project goals[7]. Chaos Report of 2009 (The Standish Group 2009), finds that only 32% of the software projects are successful (i.e., delivered on time, within budget & with quality) and the trend remains the same throughout the decade from Year 2000 to 2009[9].

Thus, it becomes critical to have enhanced focus on timely identifications of factors that can impact project, their realistic assessment in terms of its impact on project and identifying action plans which contain their impacts. Risk Identification, assessment and tracking are usually the scope of software risk management. Software risk management helps identify all such factors which can impact project outcomes, assess their impacts and plan for addressing these factors so that their negative impacts on project outcomes can be nullified or minimized. It is important to evaluate the overall risk exposure to the project in terms of some critical dimensions like costs, time and customer satisfaction.

Due to its close linkage with project outcomes, software risk management is one of the most widely researched topics of software engineering by both researchers of software engineering and practitioners in industry. But while there have been many studies around the initial phase of risk identification, there are relatively lesser incidences of studies indicating methods for assessment of risk impacts on project outcomes.

The objective of this paper is to focus on this gap and present a method of impact assessment of risk factors in quantitative terms wherever possible. The effect of risk factors on project in quantitative terms will help in better prioritization and planning leading to more favorable project outcomes.

The paper is organized as follows: section 1 provides a brief background & overview of the current state of risks and risk management approaches, section 2 explains the risk assessment methodology, section 3 illustrates the challenges faced in risk assessment and recommendations presented in this paper, section 4 illustrates framework with a reference, and finally section 5 highlights the key differentiation and benefits of utilizing this framework.

2 Risks & Risk Management: Background and related studies

Software risks are incidents that endanger a successful development process leading to wrong or inadequate software operation, software rework, implementation difficulty, delay or uncertainty [7]. In one of the pioneering studies done by Boehm identified following 10 key risks faced by software projects. After this pioneering study by Boehm, more studies have been done to identify the various risks sources and critical risks that can impact the outcomes of project. These studies were done by eminent scholars in this area and these studies have focused on identifying latest risk factors based on extensive surveys done on experiences of project managers and IT projects. It has helped identify checklists for risks and their categorization across key dimensions like people, management, business, technology, tools etc. These studies have also lead to identification of context specific risk factors such as those applicable in off shoring /outsourcing models, risk factors from different perspectives – customer side or vendor side perspective. These checklists are useful as they help identify potential risks common to new projects at their start; however it is also worthwhile to note that software risks lists and their impact factors have changed over time owing to changing contexts, processes and evolution of technologies as well as changing cultural dynamics. Some process based risk management frameworks are the Software Engineering Institute (SEI) Technical Report [2], the capability maturity model integration, CMMI Institute (CMMI 2010), and PMBOK formulated by Project Management Institute. As per PMBOK and most other process based models of risk management, key processes associated with risk management are risk assessment & risk control. Risk assessment includes risk identification, qualitative and quantitative analysis and prioritization. Risk control includes risk management planning, risk monitoring and risk resolution.

Further risk management planning strategies are varied including those of transferring the risk, avoiding the risk, reducing the negative effect of risk or at times accepting some of the risk. These are different choices depending on project situations and types of risks (positive or negative risks)

However in this paper we shall focus more on risk impact assessment, its analysis and integration to risk management framework to allow for better evaluation of impacts of risks on project outcomes.

3 Risk Impact Assessment

A risk assessment method is used to quantify the degree of importance of software risks to project performance. Most commonly risk impact is measured in terms of exposure to specific factors that present a threat to achieving the expected outcomes of a project. On this basis, risk in software projects is usually defined as the probability-weighted impact of an event on a project. Risk exposure is usually measured in dollars or time in software projects for effort/cost and schedule linked risks respectively.

Risk Exposure (Risk Event) = Probability (Risk Event) * Loss (Risk Event)

For a project facing a number of risk events/factors, the individual impacts for each risk event can be calculated by assessing their individual probabilities of risk factors and associated costs. A summation of these risk impact values derived from all risk events facing the project will provide overall risk exposure of the project.

Risk Exposure = Σ risk events [Probability (Risk Event) * Loss (Risk Event)]

The project manager attempts to identify all potential risk events at the start of the project. The risk exposure for each event is then estimated and the exposures are quantified. Risk mitigation strategies/contingency plans are developed to control and manage the risks or reduce their risk exposures. Realization of a risk is often recognized through the onset of a predefined risk trigger or the reaching of a predetermined risk threshold, at which time predefined contingency plans are activated to minimize the impact

Three well-known software risk assessment methods in the literature are PRA, SRE, SERIM and DoD Guide.

Probabilistic Risk Assessment (PRA) is a systematic and comprehensive methodology to evaluate risks associated with a complex engineered technological entity (such as an airliner or a nuclear power plant). In a PRA, risk is characterized by two quantities:

 the magnitude (severity) of the possible adverse consequence(s), and  the likelihood (probability) of occurrence of each consequence.

Consequences are expressed numerically (e.g., the number of people potentially hurt or killed) and their likelihoods of occurrence are expressed as probabilities or frequencies (i.e., the number of occurrences or the probability of occurrence per unit time). The total risk is the expected loss: the sum of the products of the consequences multiplied by their probabilities. However this model is criticized for it cannot account for the indirect, non-linear, and feedback relationships that characterize many accidents in complex systems

The Software Risk Evaluation (SRE) method was developed by the Software Engineering Institute (SEI) to identify, analyze, and develop mitigation strategies for controlling risks. SRE Method, describes the rating of the probability of occurrence on a scale of one to three, while the impact was defined by four components: cost, schedule, support and technical performance

Software Engineering Risk Management (SERIM) is used to predict risks and to provide corrective action in the software development phase. The SERIM identified 10 software risks and came up with 81 related questions that contained metrics to measure the software risks. The relationship between software risks were identified using three of the risk elements: cost, schedule and technical performance

The Risk Management Guide for DOD Acquisition (2003), released by the Secretary of Defense and the Defense Acquisition University (DAU), proposed a risk process model for how an organization identifies, assesses, and manages risks during software development. In this method, five levels were identified to assess the probability of occurrence of software risks namely, ‘‘Remote’’, ‘‘Unlikely’’, ‘‘Likely’’, ‘‘Highly likely’’ and ‘‘Near certainty’’. The impact of software risks represents the degree of influence on project performance. The types of impact of software risks on project performance in the DoD Guide includes technical performance, cost, schedule and team

RiskAoA is United States Department of Defense (USDoD) project Risk Management tool, allowing the instantaneous review of portfolio (see Project Portfolio Management), proposal or alternatives Risk. RiskAoA is proprietary to the United States Government and is generally used for evaluation of alternatives and capital planning decisions.

Of all the risk management frameworks, the RiskAoA seems most advanced from its brief description available publicly, however it is proprietary and not available for common use.

3.1

Challenges in Risk Assessment and Recommendations in

Proposed framework

The simple approach of quantifying risk impact by the use of probability of risk event and risk costs, though useful as starting base for identifying a project’s risk exposure is beset with few challenges and problems which need to be addressed to arrive at a more realistic quantification of risk exposure facing the project. This section presents some of the common challenges and recommendations from the framework proposed in this paper.

3.1.1 Determining combined risk exposure of project in view of

overlapping or linked risk events.

In situations where most of the risk events are completely independent, combined risk exposure can be calculated by adding the risk exposure of individual risk events. However in practical situations not all risk events are independent and simple summation of all risk exposures to determine overall risk exposure

will give an erroneous picture. Thus challenge is in determining ways to combine risk impacts from individual risk events and arrive at overall project risk exposure that reflects the project situation more accurately. This kind of relationship is also explored in ProRisk framework [6], however this paper augments it further by illustrating additional risk relationship types. To explain the linkages between risk events, some illustrative cases are given as below:-

 Some risk events may be overlapping in risk effects

For example: - A risk event linked to loss of key personnel of project may lead to deficient design and further defects from final testing phase. Thus if three risk items are defined viz. Loss of Key personnel mid-way in project, decrease in problem fixing rate and defects from final testing phase, the risk exposure due to each of these risk events may vary, but since due to their overlapping nature, further evaluation is needed to understand the extent of linkage between these three risks

Risk Exposure due to above risks is to be calculated as sum of individual risk exposures and adjusted by effect of common risk exposure due to their overlapping effect.

Risk Exposure = Σ Risk Exposure (all overlapping risks) – Common risk exposure of overlapping risks.

 Some risk events may be related and may get subsumed in some other risks. For example: In the event of two risks, one linked to unavailability of test equipment in lab and other linked to

availability of a specific lab equipment, it is apparent that this case unavailability of lab is primary risk which if materializes renders the second risk linked to unavailability of a test equipment inconsequential. Loss of key personnel Decrease in problem-fixing rate More defects from final testing phase

In the above case, total risk exposure due to risks falling in this category could be taken as the risk exposure due to the main container risk. However if the main container risk is managed and its risk exposure decreases to the level of individual contained risk events, then it is better to discard this relationship and start treating these risks as individual risk events for the purpose of calculating their combined risk exposure.

Risk Exposure = Probability (Container Risk) * Risk Cost (Container Risk)

Assuming that both risk cost and probability of container risk is higher than that of contained risks.

 Some risk events may be dependent on other risk events as pre-conditions. For example risk events customer raises on-site support requirement and on-site support travel impeded due to visa regulations have conditional relationship. The risk of on-site support travel being impacted due to visa regulations arises only if the earlier risk of customer raising on-site support requirements materializes.

In the above case, risk exposure of contingent risks should be determined using the principles of conditional probabilities i.e probability of contingent risk is conditional to its pre-condition risk.

Risk Exposure = Probability (Conditional Risk |Pre-condition Risk)*Risk Costs (Conditional Risk)

Recommendation in proposed framework in this paper is to evaluate risks linkages by creating risk and effects dependency models/graphs (as suggested in few examples above). Modeling multiple risk

Unavailability

of lab

Customer raises on-site support

requirement

on-site support impeded by visa

regulations

events in such graphs, would help cluster risks in few specific categories and deal with their risk impacts accordingly. The possible categories are:-

 Fully independent risks – A set of orthogonal risks whose risk impacts can simply be added,

 Partially overlapping risks– These have partially overlapping effects and for overall risk exposure the risk exposure due to interfering portion should be quantified and not double accounted.  Container risks – These risks are umbrella risk which if materialized will render many other

smaller related risks inapplicable. Thus for project risk exposure purposes, only the larger of the risk effect of main container risk should be considered. As the exposure of umbrella risk reduces, the individual risk effects of risks contained within umbrella risks should be monitored and should separately quantified.

 Conditional risks – These risks are contingent to materialization of some other risk events. Thus

their risk exposure is to be determined based on conditional probabilities of the risk event.

3.1.2 Distinction between current and incurred risk exposure in

projects

Typically risk exposure is derived from magnitude of risk events facing the project and is combined to provide the current risk exposure. The assessment of risk exposure is generally done for evaluation of impact due to risks yet to be faced by projects, but the evaluation of costs incurred due to risks to which projects was already exposed needs to be quantified and tracked too.

Recommendation in proposed framework includes two metrics to be concurrently monitored in the framework by projects managers.

 Future Risk Exposure (FRE): Indicates the total probabilistically evaluated risks which may yet materialize in project.

 Materialized Risk Exposure (MRE): Indicates the impact of risks already materialized and impacted the project. This may be on account of

o Risks which were accepted in the project risk and their risk impact has added to project costs

o Risks which were handled through risk mitigation or contingency plan. The cost of risk mitigation and contingency efforts needs to be included as part of MRE.

The proposed framework suggests continuous tracking of both these metrics. Initially all risks would be indicated as future risk exposure (FRE) and gradually as risks materialize, materialized risk exposure (MRE) would start increasing and FRE should decrease.

Total Project Risk Exposure = MRE + FRE

The above risk exposure indicates a more comprehensive view of impact of all risks current and past faced by project.

The framework proposes continuous evaluation of these metrices on weekly (or as necessary) for the project, and ensure that as project progresses future risk exposure is on decline as that shall highlight the efficacy of risk management strategies undertaken in project. MRE is likely to increase as project progresses; however it should be project manager’s endeavor to minimize MRE to minimize additional actual costs incurred in project due to risks management.

3.1.3 Integrating multiple impact dimensions of risks

The main dimensions for management of a software development are with respect to its budget (cost), availability (schedule), quality, and some other key intangibles like customer satisfaction, business confidence. Likewise risks also have impacts along these multiple dimensions, but the magnitude or severity of impact may vary across these dimensions. Most of the risk assessment done is usually with respect to impact being measured in loss or additional cost due to risk. This approach makes the impact linked to schedule delays, or customer satisfaction becoming obscure and less prominent as compared to risks whose impact can be quantified in terms of cost.

Recommendation in proposed framework: In order to have a balanced perspective of risk impacts, the framework proposes that all risk be evaluated in terms of cost, schedule and intangible impacts such as

customer satisfaction. Risk assessment from different perspectives is also explored in ProRisk framework [6], however it does not make it clear whether each risk needs to be categorized on only one dimension. The framework here allows a risk to be categorized along multiple dimensions and risk exposure from each dimension is evaluated separately. A risk may be evaluated to have impact on either one or more of these dimensions, and it should be noted. This makes it imperative that risk exposure will be measured in different units for these different dimensions.

 Risk impact on schedule – Risk Exposure or Impact should be measured in terms of days

 Risk impact on budget – Risk exposure or impact will be measured in terms of cost, as per usual

assessment methods

 Risk impact on intangibles (quality/customer satisfaction) – Risk exposure cannot be quantified in

numeric scales and hence a Likert scale should be utilized ranking the risks on five-point or seven-point scale (eg: Very Low, Low, Med, High, Very High, Extra High)

The distinction of risk impacts across multiple dimensions helps give a more holistic risk assessment and understanding to project management team to address the right areas for risks carrying both severe impacts by combining quantitative and qualitative aspects.

4 Proposed Risk Assessment Framework

The proposed risk assessment framework with recommendations given in earlier section has been implemented using a simple excel based spreadsheet. The sheet is devised to be a comprehensive risk management sheet and is utilized for:

 Risk Identifications: All risks being faced or already faced by project are identified and captured in the sheet as risk events. Some key risks may result due to several constituent risk factors and thus all sub-risk factors are listed as risk causes.

 Risk Impact Key Area Categorization: As per recommendations of framework and to highlight all possible risk impacts, each risk is assessed for key risk dimension it may be impacting and those are listed (eg: cost, cost and schedule, cost and quality etc).

 Inter-risks Relationship Determination: All risk causes need to be examined with respect to other risks to determine inter-risk relationships. The risks need to be categorized as ‘Independent’, ‘Container’, ‘Contained’, ‘Overlapping’, ‘Conditional’ and ‘Pre-condition’ risks. It is important to identify which risks are overlapping with which other risks, also clearly note which risks are pre-condition to other pre-conditional risks. Being a simple excel based tool, this implies that such

relationships are to be noted manually in sheet and this would be later used for determining combined risk exposure.

 Risk Probability Determination: Subjective assessment of each risk cause is recommended to be done by relevant SMEs (Subject Matter Expert). Delphi technique is recommended to get a more accurate assessment of probability of each risk situation occurring in project.

 Risk Impact Assessment: Individual risks are evaluated for applicable risk dimension (cost, delay etc) and their impact is to be quantified in monetary terms for cost or days for delay. For

intangible aspects, qualitative scale is used. For quantitative impacts, based on risk linkages or inter-risks relationships determined earlier, combined risk exposure is calculated.

 Continuous Evaluation of Project Risk Exposure: Combined risk exposure is differentiated between incurred risk costs or impact and future risk exposure. This differentiation is used further to define risk management strategies for high and medium range risks in project.

A reference excel based framework with proposed recommendations to arrive at more realistic and holistic risk assessment is provided.

Sample_Risk_Assess ment

5 Summary & Conclusion

The main objective of this paper was to focus on the area of risk impact assessment among all the areas of risk management. This area is not dealt so much in detail in the area of software risk management as compared to other area of risk identification and risk management strategies. While focusing on risk impact assessment, the paper deals with multiple challenges at conceptual level (detailed in section 3.1 above) and also provides a sample framework /template to manage these challenges at practical level (refer section

4

 Manage risks more holistically by providing flexibility to assess risk impacts on multiple dimensions (cost, schedule, customer satisfaction, quality etc.).

 Evaluate risk impact/exposure on project quantitatively by using classical probabilistic impact based assessment i.e PRA (Probabilistic Risk Assessment) method

 Enhances above method by allowing exploration of inter-risk relationships/dependencies and adjusting risk exposure as per these relationships. Adjusted risk exposure is net risk costs or impact facing by project.

 Continuous risk assessment and clear distinction between already materialized risk exposure (MRE) and futuristic (yet to be borne) risk exposure (MRE). It helps in identifying correct risk management strategies to minimize future risk exposure to project.

For practical evaluation of this framework, the framework was utilized for risk assessment in complex, fixed price software development projects with a scale of budget in millions Euros and duration of over one year as well, and managers using the framework found the assessment of risk exposure more objective, accurate and also helped give clear indication to management on

efficacy of risk management strategies deployed in project by continuously evaluating risk impacts to projects.

6 References

1. Srikrishnan Sundararajan, Bhasi M., Pramod K.V., “Empirical Study of Industry Practice in Software Development Risk Management”, International Journal of Scientific and Research Publications, Volume 3, Issue 6, June 2013 1 ISSN 2250-3153

2. Marvin J. Carr, Suresh L. Konda, Ira Monarch, F. Carol Ulrich, Clay F. Walker, “Taxonomy –Based Risk Identification”, CMU/SEI-93-TR-006, Pittsburgh, PA, Software Engineering Institute, 1993, [SEI-93]

3. Barry W. Boehm, “Tutorial: Software Risk Management”, Les Alamitos, CA, IEEE Computer Society,1989, [Boehm-89]

4. Y.H. Kwak , J. Stoddard, “Project risk management: lessons learned from software development environment”, Technovation 24 (2004) 915–920

5. Project Management Institute, “A Guide to the Project Management Body of Knowledge (PMBoK), 3rd Ed. ANSI/PMI 99-001-2004”, PMI, Newton Square, PA, 2004.

6. Geoffrey G. Roy,”A Risk Management Framework for Software Engineering Practice” 7. Boehm B., “Software Risk Management: Principles and Practices”. IEEE Software, Vol. 8

(1), 1991, pp. 32-41.

8. Wallace L. and Keil M., “Software Project Risks and Their Effect on Outcomes. Communications of the ACM Vol. 47(4), 2004, pp. 68-73