Chapter 4: Model Development (Action Research Cycle One)
4.5 E VALUATION
4.6.2 Considerations for Practice
Although the team were unable to test the whole model in its entirety, for the experience items where quality data was available, a good correlation to churn was established for most of these. This propensity to churn appears to increase when the experience items are combined giving optimism to the aggregation functionality with the model. Data quality was perhaps the biggest obstacle the team encountered in constructing a reasonable model. Only 6 of the 13 originally identified items were available and the team felt that the ones available did not provide the strongest impact. Also the 6 chose items provided a crude proxy for the phenomenon under investigation. This gave the team confidence that with better data quality and more appropriate proxies, the model would be significantly improved. Frequently, manner organisations talk about data quality problems, issues and concerns, but not many do anything about it (Solomon, 2005). However the lack of quality data noted above has result in Telco investing in a number of network data capture and analysis tools and enhanced data aggregation capability through a re-invigorated data warehouse, which would aid future cycles
of research with experience items that the team were unable to assess in the second cycle.
Most large businesses run a huge number of legacy systems collecting data in different formats. This data is frequently not collected with analysis in mind and therefore key attributes can be missing. Data fusion across different legacy systems can be extremely difficult and often requires a lot of manual intervention and data cleaning (Nauck et al., 2006). These were significant issues faced by the teams during this Action Research Cycle. In addition, the 6 out of 14 data items that were available for analysis were not the closest proxies of the real information the team were hoping to collect.
This is often the case with modelling systems, however the proxies available, were more detached from the real indicator than the team had hoped for. For example, with coverage the team were keen to obtain a really good measure of a customer experience with coverage. Dropped call data would have been better, but this was not attainable at the time of the work. Instead the team had to use a measure that illustrated whether a customer had a 90 % or higher chance of having good indoor coverage. This gave us a very crude and binary: >90% chance of indoor 3G coverage = good coverage; <90% chance of indoor 3G coverage – bad coverage. Firstly there is the accuracy of the 90% chance statement and secondly the team don‟t get a continuum or range of excellent coverage through to mediocre coverage and onto bad coverage, which would be more realistic. Put another way the range under not good coverage is much too broad. This said, the impact of coverage appears to be so strong that its impact on experience and thus loyalty and churn was felt in the analysis conducted.
The cost of radically improving data management and ensuring all relevant data items are maintained is not an insignificant amount of work, energy and change of approach, which Telco should not underestimate. This central issue has been a perennial bug bear of information systems design, operation and maintenance for
a long time, with most solutions pointing to hard work and a determined and tenacious attitude to resolving data issues (Wang, Storey & Firth, 2002).
Due to a lack of expertise and tools, data analysis is often done in a too simple or naïve way. Linear models are the most frequently used analysis methods, because they are easy to understand (Nauck et al., 2006). This was experienced with Telco, with team members‟ “eyes glazing over” during the debate over appropriate analysis methods. It was explained that linear models assume that linearity assumes normally distributed values and that variables are independent of each other. When considering the data available to the study, the theory on churn behaviour and the experience of the team members, it was highly likely that variables would be inter-related and the relationship with churn would not be a linear one.
It was evidently clear from this short study that tackling experience in this way requires a concerted effort on behalf of Telco in order to gain traction. It was the first time a cross functional team had been assembled to focus on Customer Experience and retention. Following the end of this study, Telco decided to set up a customer experience / churn programme headed up by the Director of Customer Management. This illustrates a belief that there were gains to be had by focusing on Customer Experience, but also that a senior director with responsibility for customer management needed to be formally engaged and tasked with bringing about improvement. The director was keen to enlist the assistance of the researcher to help shape and drive the programme of work.
In addition to the customer experience and churn programme, other projects have been initiated, as a result of the work done. The network team in Telco are investing heavily in network probes that will provide better network experience data and also investment is being made into the data warehousing infrastructure in a bid to make more implicit data available to systems and users that may need it. The study has also given Telco confidence that improving Customer Experience is an attainable and profitable goal. Business programmes have been initiated, aimed
at improving organisational alignment, ensuring staff take responsibility for the experience customers receive.
The team started to think about how it would present this information in the context of analysis in the preparation to proactively call a customer and within the context of having a conversation with a customer in real time. The presentation of the information was almost as important as the analysis of good or bad experience within the overall CEM model. There is always a tremendous amount of information during an interaction with a customer and part of the success of any solution would be an ability to present the required information in an informative but easy to digest manner. This formed the basis of the challenge for the next Action Research Cycle Phase.
4.7 Summary
This chapter describes the development of a model for assessing and monitoring Customer Experience. Driven by the request of the support organisation Telco, a set of per-determined JD Power experience categories were utilised. Using implicit data, the categories were decomposed into experience items (items that provided proxies of experience), aggregated and considered over time to establish profiles. Analysis across 2 separate customer data samples, demonstrated a positive correlation between the poor aggregate experiences and a propensity to churn. Although the correlations were weak, they were statistically significant and provided the encouragement Telco needed to progress. The weak correlations were impressive given that the churn data will be affected by many other factors, such as branding awareness, advertising and social network factors. In addition when consideration is given to the fact that data quality issues prevented the team from having the best possible data and indicators, the team were surprised and encouraged that the experience indicators can be heard above this noise. Attention now turns to how this information can be usefully presented to the front-line teams.