Evaluation

The table above shows that various complementary data collection methods were employed in an attempt to triangulate the results and increase the validity of the findings; the intention being that the limitations of different methods would be counteracted by others, producing a stronger end result.

Action Research Cycle One Evaluation

The evaluation criteria for the success of cycle one centre around:

Data availability – examining which data elements were available or accessible to allow comprehensive analysis to take place.

Experience and Churn relationship – Analysing whether a relationship between aggregate experience and churn could be established?

Desire of Telco – Review whether the results would generate enough interest in Telco to continue the study.

The validation of the customer experience concepts were to be arrived at via a process of data triangulation. This process synthesised data from the customer experience literature, the JD Power Survey (both areas previously discussed) and the thoughts from real customers, with the aim of further validating the customer experience items and their relative importance. This triangulation along with iterative discussions and input within the team ensured the interpretation of the results would be grounded, thus providing sound foundations upon which to build a Customer Experience Model.

Table 3-5 – Details of Analysis for Action Research Cycle One

Quantitative Analysis Details Qualitative Analysis Details The analysis is based on two datasets

6,529 customers and 8,145.

The samples contained 50% of customers who churned and 50% who stayed with the operator.

The churn or stay date was the end Dec 2008

The samples contain customers on 12 or 18 month contracts, with the volumes mirror the split across the entire based (c. 33% and 67% respectively).

The analysis period covered data from month 4 of the new or upgraded contract to months 9 (for 12 month contracts) and from month 9 to month 15 (for 18 month contracts). This was because we wanted to capture the in-life experience.

All other aspects of the data mirror the entire customer population base.

The survey of real customers involved a 5 minute face to face questionnaire interview with Telco customers as they were entering or leaving Telco retail stores.

This was conducted simultaneously across 4 different geographical locations. A sample of 94 customers were surveyed, with a 59% to 41% male to female gender split respectively and a spread of age ranges from 18 to 65.

Telco advised that this demographic split was representative of their total customer base.

Action Research Cycle Two Evaluation

The evaluation criteria for the success of cycle two centre on:

Match to existing systems – Reviewing whether the prototype solution had the same look and feel as existing front-line system and presented useful experience information.

Front-line and team feedback – Considering if this solution fitted into existing business processes and supported front-line employees in their roles.

Team feedback – Examining whether the prototype articulation of the Customer Experience Framework has practical utility.

General understanding of the business case – Assessing the costings for a production solution and determining what the expected return on investment could be..

Formally, prototype evaluations can be conducted in a summative manner, where the design is evaluated after it is complete or nearly complete, or in a formative way, where the evaluation beings early as is continually conducted the development process (Hix & Hartson, 1993).The idea with formative evaluation is that it provides time for modifications to be made to the design, and this approach is sometimes mistakenly thought to be not a rigorous and too informal when compared with summative approaches. However the iterative nature of the development and the close collaborative working environment meant that a formative approach had a greater chance of leading to a prototype that met with Telco‟s requirements. In addition formative evaluation necessitates the collection of data, which is used to develop an informed view of the work. Generically this can take the form of objective data such as performance bench-marking and testing against pre-defined use cases or subjective data represented by the opinions of key stakeholders. This said, prototyping can serve as vehicle for organisational learning, with the development process serving as a means of exploring the meanings and purposes behind action (Checkland & Holwell, 1998).

3.6 Summary

Chapter 3 ascertained that there are a number of valid theoretical meta-models /world views that can be adopted to form the philosophical core of the research methodology. The demand for practical improvement, both from the researcher and Telco meant that pragmatism was deemed the world view that aligned best

with this desire and best suited the type of organisation Telco had developed into and the environment it operated in. In analysing the research aim, it seemed that the study would benefit from elements of both quantitative and qualitative approaches and therefore a mixed methods approach was adopted. Despite the criticism of Action Research, much of which refers to poor research practice in general, and with the application of sound guidelines for good Action Research conduct, Action Research suited the pragmatic and mixed methods approach and was selected as the method to use.

The study followed a Canonical style of Action Research reviewing carefully the diagnosis, action planning, action taking, evaluation and learning aspects. The research design consisted of the potential for 2 Action Research Cycles. This first cycle was designed around the quantitative analysis of churn data for c. 6,500 and c. 8,000 customers to ascertain linkage between experience indicators and churn. It was also devised to include qualitative output from Telco customers, concerning their attitudes to experience to help provide triangulation and confidence that the correct experience categories and items were being selected. For the Second Action Research Cycle, a prototype was envisaged, to help hone ideas and thoughts from the First cycle and provide a realistic solution in depicting how a solution could be put together. This cycle would have additional validation through feedback captured during interviews with Senior Managers.

Action Research is presented as offering an appropriate and serious alternative to the hypothesis testing approach favoured by natural scientists. Action researchers however must endeavour to ensure the research process is recoverable. The chapter maintains that if the research is well organised, researchers should be confident that the results can stand up to plausibility claims and critical suggestions that the results are merely anecdotal.

Chapter 4: Model Development (Action