Process Improvement Reviews
Why your team should have regular
Process Improvement Reviews
Leanne Howard, Agile Practices Consultant
Planit Software Testing
Abstract
We all should be continuously challenging ourselves to make
improvements in efficiency and our effectiveness. This article discusses
different processes that can be used to do this.
Planit Software Testing www.planit.net.au www.planittesting.co.nz www.planittesting.co.uk
With the increase in popularity of Agile, one of the ceremonies undertaken by many Agile teams is the retrospective. This regular meeting investigates what the team is doing well and thus should continue doing, as well as which areas could stand to be improved, prompting the team to make suggestions on how they think these areas could be improved.
Even with retrospectives in place, continuous improvement is not guaranteed. For instance, the team may try to make too much change at once without giving the new ideas time to become embedded, or suggested improvements may never be implemented if no-one takes ownership of that initiative.
In traditional projects this is even worse as the Post Implementation Review (PIR) does not happen until the end of the project when it is too late to implement improvements. In such instances, the team often gets disbanded which means the lessons learnt may never be applied in the following project and the same bad or inefficient practices can persist.
Often when you are heavily involved in the day-to-day delivery of a project it is difficult to see the “wood for the trees”, so an independent viewpoint is helpful. Also, if you move from project to project or manage a number of projects it is often difficult to make available the time needed to do the root cause analysis required in order to look at more efficient ways to work. If you have been working for the same company or even within same domain, there may be better ways to do activities or new techniques to use that you may not know about. An outside person can also give additional credibility to ideas that you have had but have not had sufficient resources in order to change or even document standard policy. Once you have established that a Process Improvement Assessment (PIA) is required, you must then discern which option is the right choice for your situation.
But first, as with any project, it is important to understand your requirements and their associated priorities. You must then work out what the costs and benefits will be for each activity, being sure to account for indirect costs and benefits.
Assessing Costs
Effort and costs can vary depending on:
The assessing organisations costs – i.e. their charge rate
The size and scope of the assessment i.e. is it an organisation, sector of the business, program or project?
The number of projects typically running, as it is important to assess against a percentage of live projects
The number and location of interviews that need to be conducted
Whether it is a formal or informal assessment Very often only direct costs and benefits are
considered, especially given the potential difficulty in quantifying indirect costs. Nonetheless, these should be assessed as their impact can be substantial. Direct Costs
Monetary
Work hours
Training and education
External consultancy
Assessing Benefits
Like costs, benefits are also direct or indirect, with indirect benefits often being more substantial than their more directly associated counterparts. Direct Benefits
Rise in productivity
Early defect detection therefore cheaper to fix
Fewer costly issues in production Indirect Benefits
Improved staff experiences, morale and motivation leading to a better quality work
Increased customer loyalty
Better opportunities for employee movement within projects
Improved working environments
Recognises testing as a profession and integrated in the development process
When examining the process improvement options available, it is important to understand the costs, advantages and disadvantages of each as well as their differing level of formality. These factors should be matched with your organisational structure and needs. In order to help decide which Process Improvement approach to follow, a summary of each one has been included below.
Test Maturity Model Integration (TMMi)
The Test Maturity Model Integration (TMMi) is the only process improvement option that provides a certificate of maturity. It aims to introduce a structured / controlled set of test processes, moving to a level of optimisation through increased rigour.
Developed by the TMMi Foundation as a guideline and reference framework for test process improvement, this model is positioned to complement the Capability Maturity Model Integration (CMMi), being aligned with v1.3.
TMMi has a staged architecture for process improvement, with organisations passing through a number of
stages/levels as its testing process evolves from one that is ad-hoc and unmanaged to one that is managed, defined, measured, and optimised. Achieving each stage
ensures that an adequate improvement has been laid as a foundation for the next stage.
The internal structure of the TMMi is rich in testing practices that can be learned and applied in a systematic way to support a quality testing process that improves in incremental steps.
There are five levels in the TMMi Model that prescribe a maturity hierarchy and an evolutionary path to test process improvement. Each level has a set of process areas that an organisation must focus on to achieve maturity at that level. Experience has shown that organisations do their best when they focus their test improvement process efforts on a manageable number of process areas at a time, and that those areas require increasing sophistication as the organisation evolves.
Because each maturity level forms a necessary foundation for the next level, trying to skip a maturity level is usually counterproductive. At the same time, it must be
recognised that test process improvement efforts should focus on the needs of the organisation in the context of its business environment and the process areas at higher maturity levels may better address the current needs of an organisation or project.
Benefits
Improved organisational effectiveness and efficiency
Improved product quality
Improved testing productivity – potentially leading to shorter delivery time frames
Suited to ANY development lifecycle model
Shifted focus from defect detection to defect prevention
There are two types of TMMi assessment:
Formal Assessments
Formal assessments must be conducted by an assessment team led by an accredited lead assessor and at least one other accredited assessor.
Formal assessments require a strict level of evidence for the achievement of specific and generic goals in the relevant TMMi process areas. One mandatory point of evidence is in the form of staff interviews, which must be corroborated with the findings from the document study. One of the results of formal assessment is a full gap analysis showing the strengths and weaknesses of an organisation against the TMMi model. This gap analysis
can be used as the basis for future improvement projects. A formal TMMi assessment typically acknowledges the following phases: 1. Planning 2. Preparation 3. Interviews 4. Reporting
Informal Assessments
Alternatively, informal assessments can be conducted, following a less rigorous process than is required for formal assessments, resulting in faster and cheaper delivery, but also one that is less precise. Informal assessments are led by an experienced assessor, designed as an initial indicative view and ‘quick check’ to evaluate the current state of the test process against TMMi.
Implementing TMMi
First and foremost, TMMi is a list of best practices or a description of a mature test process. TMMi does not offer a standard approach to a change program in an
organisation.
To support the implementation of models such as CMMI, the Software Engineering Institute (SEI) has developed a model for change processes IDEAL. This model has also proved to be very useful when implementing TMMi. IDEAL offers an extensive and practical reference standard for change processes and also shows what needs to be done when implementing TMMi improvements in an organisation. The model contains a five phase improvement cycle as shown below:
Phase Goal
Initiating Establishing the initial improvement infrastructure for a successful improvement process. Diagnosing Determining the organisation’s current state as opposed to what it wants to achieve. Establishing Planning and specifying how the chosen situation will be established.
Organisations are free to choose the appropriate improvement approach for their implementation of TMMi.
In addition to IDEAL, there are several other models for the implementation of process improvement. In general these models are based on Edward Deming’s plan-do-check-act cycle. The Deming cycle starts with making a plan that determines the improvement goals and how they will be achieved (plan). Then the improvements are implemented (do) and it is determined whether the planned advantages have been achieved (check). Based on the results of this assessment, further actions are taken as needed (act).
Each of these change management processes can also be used with any of the Process Improvement activities outlined in this whitepaper, or other alternatives that may not be covered.
Test Process Improvement (TPI)
Test Process Improvement (TPI) acts as an independent audit of current testing practices versus testing best practices to assist organisations in bolstering the likelihood of project success and to improve productivity and cost efficiencies in testing.
The key areas covered in a TPI include but are not limited to:
Examining the strategy and approach to testing
Cataloguing the roles, responsibilities and capabilities of the existing test team
Analysing of training needs
Reviewing existing test procedures and standards for testing
Reviewing existing test assets
Assessing third party vendor testing and software release management
Assessing test environment set-up, configuration and management
Reviewing regression testing and the regression test pack
Reviewing performance testing, including the use of testing tools
Evaluating test tools to support the testing process
The TPI audit model is specifically designed to measure testing processes, skills and practice across an entire organisation and their interactions with other teams throughout the project’s entire lifecycle, addressing 20 key areas. The power behind this model is to specifically identify the current stage of testing 'maturity', identifying where the organisation is headed or needs to be and then mapping activities for improving this status as well as checkpoints to ensure that the end result is achieved.
Assessment Criteria: Test Strategy Life-cycle model Communication Reporting Evaluation Low-level testing Metrics Test automation Test environment Office environment
Commitment and motivation Test functions and training Scope of Methodology Test specification techniques Static test techniques Defect management Testware Management Test process management Moment of involvement Estimating and planning This improvement review can be conducted across divisions or projects to provide a benchmark and guide organisations as to which assets are performing well and thus should be re-used. This could also be completed for one project to help them reach their milestones in an optimal manner. Part of the deliverable is a comparison graph which clearly and easily highlights areas to be improved.
An updated version of this approach has been coined by Sogeti called TPI NEXT, following the same process only with a slight change to the assessment criteria. These key areas examined are as follows:
Degree of Involvement Metrics Test Strategy Test Organisation Communication Reporting Test Case Design Test Tools Stakeholder Commitment Defect Management Testware Management Methodology Practice Tester Professionalism Estimating and Planning Test Process Management Test Environment
Agile Process Improvement (API)
As the name suggests, Agile Process Improvement is focussed on Agile teams or organisations moving to Agile. It focuses on the team interactions and
collaboration, product quality and the sharing of testing responsibility amongst the whole team.
The key areas to be covered in an API include but are not limited to:
Examining the strategy and approach to Agile
Planning at Release and Iteration level
Cataloguing the roles, responsibilities and capabilities of the existing Agile team
Team collaboration and interactions, including with the Business
Analysing of training needs
Reviewing existing Agile procedures and any standards/compliance requirements for the Agile team
Reviewing requirements gathering and grooming
Reviewing existing Agile assets
Assessing Agile environment set-up, configuration and management
Reviewing regression testing, including automation capability
Reviewing non-functional testing, including the
the framework/methodology used by the organisation. It highlights areas of potential improvement, steps required to achieve these improvements in productivity,
effectiveness and cost efficiency, which are all well known benefits for transforming to agile.
It is surprising to note that as many organisations struggle with their agile transformation that there are very few recognised process improvement frameworks for agile specifically. Those that are available focus on the developer and their activities. Planit’s API is unique in that it has been designed to measure Agile processes, skills and practice across an entire project lifecycle, including the Business interaction which is often one of the key challenges.
It assesses the whole team collaboration and moving to a cross functional team. There is an emphasis on whole team ownership of quality and what activities the team members can complete in order to optimise the benefits of agile and deliver product that gives the business value. The API is designed to allow Planit to get to know your organisation in more detail, which then enables us to 0 2 4 6 8 10 12 14 16 18 20 22 24
Score
Key Areas
Project X vs Best Practice
Industry Best Practice Project X
Test Process Health Check
A Test Process Health Check similar to the Test Process Improvement only less formally, utilising a questionnaire that focusses solely on the testing team. The Health Check is typically shorter in duration and is best suited to smaller organisations or those who do not require highly formalised reviews, allowing them to achieve and implement “quick wins”.
The Health Check consists of two different questionnaires; one focussed at the test management level and the other at the test analysis level.
Based on the required scope, the topics covered in the Test Process Health Check may include:
Test Framework Risk Management Test Reporting Test Environments Test Automation Test Estimation Test Preparation Test Execution Defect Management Test Metrics
Testing within the SDLC Regression Testing
Test Career Path & Organisation Change & Release Management Test Improvement Process The Health Check process and activities are split into four distinct phases:
Initiation and preparation
Interviews
Review of test artefacts
Completion and reporting
Implementing Improvements
Regardless of which areas you are targeting for improvement, it is important that you allocate the right resources in order to realise the true benefits. You are about to embark on a change management programme which needs to be managed and supported as such. Early successes through the implementation of “quick wins” will help to gain momentum although introducing too much change at once that cannot be absorbed by the team can be disastrous and derail the good before it has ever begun.
Change Curve
Enterprise change, or even change at the project level, evokes emotional reactions that may be categorised using Elizabeth Kubler-Ross’s (1970) Model of Emotional Reaction to Change. Anger Denial Shock Bargining Depression Acceptance Experimentation Discovery Integration S e lf e st e e m Time
While death and dying is the ultimate trauma for many people, similar emotional upsets can be experienced when dealing with many of life's challenges. This is especially the case if confronting something difficult for the first time or if the challenge happens to threaten an area of psychological weakness, which we all possess in different ways.
Of course as individuals we all react differently and this will also depend on the size of the change. Often if people are included in the definition of the change, plus if they can see the benefits for them and ultimately the organisation, which are articulated consistently, it is more likely that you will get buy-in or at least acceptance. What you are really aiming for is enthusiasm for the change, introducing a culture of continual process improvement where challenging ineffective and inefficient activities is commonplace.
If you are trying to implement these new practices within teams’ particularly new ones, it is also important that you understand some team dynamics.
Team Dynamics
When looking at introducing change you also need to understand team dynamics. The Forming – Storming – Norming – Performing Model of group development was first proposed by Bruce Tuckman in 1965, who posited that these four phases are necessary and inevitable in order for the team to grow, face up to challenges, tackle problems, find solutions, plan work and
deliver results. If you understand where you team falls in this model, you will better be able to recognise the acceptance or disruption that change is going to bring to them. In summary the phases are:
Forming
In the first stages of team building, during the forming phase, individual behaviour is driven by a desire to be accepted by the others, and avoid controversy or conflict. This is a comfortable stage to be in, but the avoidance of conflict and threat means that not much actually gets done.
Storming
Groups next enter the storming stage in which different ideas compete for consideration. The team addresses issues such as what problems they are really supposed to solve, how they will function independently and together,
and what leadership model they will accept. Team members open up to each other and confront each other's ideas and perspectives.
Norming
At this stage, the team has come together with one mutual goal and plan, with all team members working towards ensuring the success of these goals. To reach this stage, some team members may have to concede their own ideas and follow team consensus in order to be a functional unit.
Performing
Teams at this stage are high-performing, able to function as a unit as they find ways to get the job done smoothly and effectively without inappropriate conflict or the need for external supervision. Such teams are characterised by interdependent team members.
Conclusion
With the increasing adoption of Agile, the concept of continuous improvement has become a regular practice for many teams, with the principles of “inspect” and “adapt” helping to improve feedback systems. Even those working on more traditional projects are adopting continuous improvement or at least regular checks on how the project is tracking with the goal of finding potential gains in efficiency.
But when deciding on how to review processes and implement improvements, it is critical that you evaluate your
organisational or project needs, as this will dictate which model will best fit your circumstances. Each of the approaches listed has its benefits, but also an associated cost.
If you understand your requirements but are unsure of which model is best for you or how you should go about
implementing this model, you should discuss this with an internal expert or experienced senior consultant rather than risk heading down the wrong path.
Planit’s highly experienced consultants are able to tailor a solution to fit within your scope, budget, timeframes and availability. By pursuing an independent assessment, you are able to maximise the benefits of this activity:
Avoiding bias and internal politics, gaining an accurate view of the true state of testing
Benefiting from extensive knowledge of industry practices based on accredited training and cross-industry experience
Adding further credibility to the recommendations
Gaining the time to dedicate to the root cause analysis
Ensuring focus on implementing improvements by having a third party dedicated to keeping the project on track Also there is no point doing this exercise of evaluating where you are currently and then making improvement suggestions, if
The key focus of testing should be to move from defect detection to defect prevention. Adhoc processes often only produce results due to the “hero” mentality of a few within your team which is not sustainable over a period of time. The aim is to move through just finding defects (debugging), checking against requirements, proving fit for purpose, good feedback on product quality to optimising the test process.
Testing accounts for at least 21% of overall project spend (Planit Testing Index 2014) and this can rise significantly if a project has to test across multiple browser and/or mobile devices combinations. Therefore if efficiencies can be made in testing this could result in large savings to the project budget or all wider coverage in testing thereby reducing risk even further.
References
TMMi Foundation
TPI and TPI Next – Sogeti
Elizabeth Kubler-Ross’s (1970) model
Software Engineering Institute (SEI) – IDEAL
Edward Deming’s plan-do-check-act cycle
Team Dynamics by Bruce Tuckman
Planit Testing Index 2014
