Dee Hook, founder of the Visa network, said that:
in years ahead, we must get beyond numbers and the language of mathematics to understand, evaluate and account for such intangibles as learning, intellectual capital, community, beliefs and principles, or the stories we tell of our tribe’s values and prosperity will be increasingly false.30
Many books on Performance Measurement often assume that all rele-vant performance data is either already available or can be easily col-lected. Unfortunately, this is mostly not the case. Even though we might have a lot of performance data in our organizations, it is often not the relevant and meaningful information we need. Instead of rely-ing on data that is available in our existrely-ing IT systems, we should first identify what we would really like to assess, and then compare it with what we already have in place.This allows us to see how close we can get with our existing indicators to what it is that we want to assess.
In many cases, the information we want is not at all, or only insuffi-ciently, available.This means we have to collect more data. One way to
Output deliverables
Core competence I Core competence II
Assessment/
indicators Assessment/
indicators Assessment/
indicators
Value drivers Core competencies Output deliverables
Figure 5.5 Assessment and indicators for strategic elements
get a good idea about how to assess performance in a particular area is to ask the people who are most closely involved in that area. Far too often we impose measures on people. Assessing performance for strategic learning means that people have to believe in the indicators and use them to inform their decision-making. Therefore, involving people (both internal and external) is critically important.
Not only should we involve people in the selection of possible indi-cators, but also in the assessment of performance and the collection of data. Many studies have shown that perceptional assessment is as reliable, if not more reliable, than archival data.31 It can provide richer insights into the real level of performance because our brain is able to comprehend performance more holistically. The way we involve people is to ask them to, for example, rank competitors, evaluate the service delivery, assess the level of relationships with different sup-pliers, etc. These assessments can take the form of numerals, or grades;
however, they can also take the form of a written assessment. Written assessments are able to capture much more information and allow us to more naturally communicate assessment outcomes. If numerals are used, these should be supplemented with at least a comments field to provide some explanatory narrative assessment in addition to a number.
Traditionally, organizations have used archival data or larger-scale surveys to access performance. Below, different data collection methods are outlined to provide some alternative ways of collecting perform-ance information and perception data:
䊏 Surveys and questionnaires provide a relatively inexpensive tool to collect data from a large pool of people who can be at different loca-tions.32This can be done via mail, e-mail, or internet. One big prob-lem with this is that there has been a huge influx of surveys over the past few years as more and more organizations require data for their non-financial indicators.The consequence of this is that it is get-ting harder to make people complete a survey. It is always a good idea to reduce the amount of time and effort required to collect per-formance data, not only for your organization, but also for your cus-tomers, employees, suppliers, etc.
It has been argued that surveys and questionnaires are one of the most difficult data collection methods to do well, as it takes a significant time to construct clearly worded questions that will result in useful and valid data. Surveys should always be pilot tested on a number of people before being used. This allows mistakes to be eliminated and clarifying questions to be added before the surveys are sent out.
110 Managing performance in an enabled learning environment
䊏 Observations allow us to collect information by observing situ-ations or activities with little or no manipulation of the environ-ment. The observer can either take the role of a passive onlooker/outsider, or can become involved in activities and, there-fore, take the role of partial or full participant observer.
The power of using observation methods is that it engages all of our senses not just our sight. It enables us to talk in and make sense of the entire experience through our nose (smell), eyes (sight), ears (hearing), mouth (taste), and body (touch). Unlike other data collection methods, observation data can provide us with a more holistic understanding of the phenomenon we’re studying.33
Observation data has the purpose of describing, which can take the format of score sheets, checklists, narrative reports, video taping or audio taping, depending on the level of detailed contextual informa-tion required.
䊏 In-depth interviews are guided conversations with people, rather than structured queries. They involve putting forward open-ended (how, why, what) questions in a conversationally friendly and non-threatening manner.34Interviews can be conducted face-to-face or via telephone or video-conference. Interviews enable us to interact directly with respondents and may result in new insights about performance and provide examples, stories, and critical inci-dents that are helpful in order to understand performance more holistically.35
䊏 Focus groups are facilitated group discussions in which partici-pants can express and share their ideas, opinions and experiences.
They provide a unique and interactive way to gather information and allow the collection of rich, qualitative information.Typically, a focus group will comprise between 5 and 20 pre-selected people who are willing to participate.
䊏 Mystery shopping is the assessment of a service by a ‘secret shopper’ posing as a client or customer. Some companies have in-house programmes, whereby the company hires its own mystery shoppers; other firms hire external suppliers to provide this service.
䊏 Peer-to-peer assessmentis the assessment of performance in which participants vote or assess each other’s performance. This can either be done openly or anonymously.This enables people to learn from each other and to consider their own performance from other people’s perspective.
Organizations have become much more creative about the way they are assessing performance. Call centres now regularly audiotape phone conversations between service agents and customers. You might recall an automated message before you are connected to a service agent notifying you that the call might be recorded for training purposes.
Many call centres now use a coach to listen to conversations and then sit down with service agents to see how they could improve the quali-tative aspects of their calls.
One organization introduced two types of carpets with sensors in the entrance hall, one red and one green. Whenever employees entered or left the building they could choose to step on either the green carpet, indicating they were happy, or on the red, indicating they were unhappy. Another organization automated their phone sys-tem to play a message once a conversation had ended, asking cus-tomers to rate their experience by pressing a button on their phone.
Many service providers, such as hotels or banks, use focus groups to identify what really matters to customers and then employ mystery shoppers to assess service levels according to the identified criteria.
There are many more fascinating ways of collecting qualitative per-formance data – for more information see, for example, the Handbook of Qualitative Research.36
It is a good idea to collect performance data using different tech-niques and methodologies. This allows organizations to contrast and compare the information gathered from the different methods. This is called ‘triangulation’. The rationale behind it is that the more informa-tion we have from as many sources as possible – which all have advantages, disadvantages, and different biases – the greater the likeli-hood that the information is reliable.
Organizations are often unaware of biases in their data. A frequently cited anecdote is the Wald story.37 Abraham Wald was a statistician during World War II who helped the air force to assess where air-planes were most vulnerable to enemy fire. The plan was to subse-quently reinforce the most vulnerable parts of the plane. Each airplane was examined for bullet holes and the areas that were disproportion-ately more often hit than others were identified. The air force nat-urally concluded that the areas with the most bullet holes should be reinforced. However, Wald made them aware of the bias in the sample. Only airplanes that returned to the base were examined and included in the analysis! This, therefore, shows that the areas with the many bullet holes have proven able to sustain enemy fire and so these planes returned safely, whereas the areas with no bullet holes might be the best to reinforce since planes hit in these areas did not return.
Triangulation, on the other hand, means that organizations collect data from different data sources (e.g. interviews with board members, middle managers, and front-line workers), use different methodologies (e.g. survey 70% and interview 30% of your suppliers), or use different people to conduct the data collection.This can reduce bias and increase reliability.