KPI
Risks
Obtaining a good KPIs may sound easy but I believe there are few challenges in front of implementation of any measurements specially KPI approach. In the following table is tried to depict about perceived challenges which any software company may face to.
No Challenge Name Description
1 Lack of Historical Data KPI can be solely useful if there is a range of data in a history of an organization despite of having very clear and preferable metrics but without historical data KPIs are hardly efficient as well as effective.
2 Various Data Sources To make KPIs understandable and more accurate, it needs various data. Usually within organization these data came from different sources, for instance: QA, project planning, IDEs, SCM and etc.
3 Normative benchmarks Normative benchmark is only good as the parameters being used for comparisons. But there are few concerns regarding this approach which are:
A) What are the benchmark criteria? B) How these criteria are shown up?
Risk Mitigation
To mitigate the risk of aforementioned items in Risk section, it has been tried in the following table to lessen the severity of each item:
Risk Name Severity Mitigation
Lack of Historical Data
Blocker To diminish the severity of this risk, we have to follow few steps which are: 1. Define the KPI perspectives
2. Narrowing down each perspective to avoid it become subjective. On the other hand everyone ought to understand each element of KPI perspectives by heart.
By defining these elements, we must spend some time to capture data according to the elements which have been defined.
Various Data Sources Medium There are few approached to address this concern such as: (1) gathering data manually or (2) purchasing some software house KPI tools like Verex or (3) develop some in-house softwares.
My recommendation is to follow the first step if your organization is below than 50 person. Then you can decide to go for second or third solution.
Normative benchmarks
Severe The solely way to overcome with this relativity smart trap is to look for objectives rather than normative metrics. So the question is this how can we achieve this?
Perspectives
There are 3 different perspectives which can be considered within each software house as KPI. These perspectives are: I. Quality
II. Innovation III. Effort
I think Verex system had a quite relevant implementation that I recommend as well. Their bird-eye view metrics is shown in following figure:
Each element of the perspective has been illustrated in following table.
KPI Perspective
Perspective Name
Metric Description
Quality Number of issues per project/ iteration/sprint
the ratio of new issues appearing vs. those being resolved. These can be taken from Bugzilla or JIRA. The obviously important pa-rameter was the trend, showing whether the number of problems was growing or being reduced. The applied 3D breakdown into the components gives an idea, where the team has the most problems or what is the technology piece causing most issues (in our case it was Hibernate and JavaScript/DHTML).
Perspective Name
Metric Description
Number of blocker and critical issues known
this gives a very good understanding of which resources are re-quired to maintain a product/project. Critical and blocker issues usually require immediate action – knowing how many are in the pipeline allows for much more realistic planning.
Average time to resolve an issue and effort required to fix all issues
this is a very good tool for budgeting and forecasting when a de-sired level of quality can be reached.
Number of test cases per each project/iteration/sprint
It is very important to encourage “writing tests as you go”, or for QA teams to generate test cases when bugs are found. Not only does it indicate the test coverage, but also the trend – coming up with a good test plan is a continuous task and new cases should appear en route.
Documentation Software is continuously documented via SCR (Software Change
Request) or user story documents. The SCR is a high-level to low-level document that in theory should be updated as function-ality evolves. Typically what happens is that the first SCR version is written and accepted, but rarely updated afterwards and quickly becomes obsolete. The number of SCRs produced is not relevant to the R&D management as they have to be provided in order to kick off implementation anyway. It is the number of changes and updates in the development process that matters as it reflects if a team documents their work.
Innovation Enhancements and improvements
Building an innovative culture into a company and not forgetting good ideas is always a challenge. That is why at any company should be decided to make the enhancements reporting as simple and as least-time consuming as possible– this was achieved by just by adding a new category in Bugzilla/JIRA. Enhancements are not rewarded in any way in order to prevent a flood of garbage (pseudo-enhancements), however they are an important aspect of project/sprint team assessments.
Enhancement in specific product/service/process areas
(e.g. GUI/front end, performance). This metric gives a good indication of which product areas require most work. Some general conclusions can also be drawn, such as general lack of resources to optimize UI response times. The risk here is that most internal users will report usability and GUI related issues, not back-end problems.
Perspective Name
Metric Description
Efforts Man hours in the project/ iteration/sprint
It is good to see weekly and/or monthly deltas, especially if the effort can be compared against and correlated with other indicators, such as the number of issues in the project.
per activity breakdown While software development budgets are usually well controlled, the direction of the spending (e.g. new features, product harden-ing, support & fixes, etc.) is usually a hazy matter. This metric immediately shows where the project effort goes to and the con-dition of each product.
Action Plan
To attain highest values of KPI, I would suggest following procedures:
I. Follow the mitigation of Risk-1 [Lack of Historical Data] in a specific time box such as 6 months(which depends on how agile your organization is to deliver different systems).
II. Then according to facts which have been captured, we put our target objectives. For instance we find out that in last 6 month our percentage of having bug in production is 20%, then we will set objective like this: we want to achieve 5% bug in production in 3 period of 6-month in harmony of: 17%, 12% and 5%.
III. At a end of each time box there shall be a report which covers an assessment like: as-is of a time and the desired to-be of the time. According to the variance if either re-planning or re-prioritizing are needed, it must be done. IV. At the end of the year there must an ROI to be done for each team. In following section an example of ROI has
been presented:
Return On Investment for 2012
The table below presents the calculation of payback period and 1st year ROI (cost oriented) for a team of 15 developers based on the following assumptions:
I. the deployment of KPI Dashboard will require 30 man days of consulting,
II. the efficiency improvement will amount to 3% (which is a very conservative assumption),
III. no additional benefits are taken into consideration (e.g. better management decisions), only developer efficiency is accounted for.
KPI tools licenses 2000
Deployment (30 man days, times consulting rate $450) 13500
Total Cost $15 500,00
Size of development team 15
Annual Salary $75 000,00
Total Annual Salary $1 125 000,00
3% efficiency improvement (per year) $33 750,00
3% efficiency improvement (per month) $2 812,50
Payback Period (months) 6
References
I. Benchmarking software development projects: The three KPIs http://www.computerworlduk.com/in-depth/ applications/1830/benchmarking-software-development-projects-three-key-kpis/?pn=1
II. Verax Systems http://www.veraxsystems.com/ III. General material from Gartner on KPIs in IT