Research protocol

By studying multiple cases it is possible to confirm a logic chain of evidence (Yin 1994; Miles and Huberman 1994). Every case should be conducted in the same way to be able to define recurring patterns and to see if relationships as proposed in advance actually exist in the companies. To conduct the cases in the same way we had to elaborate a research protocol to account for the reliability and validity of the empirical research.

Reliability refers to demonstrating that the operations of a study (for instance, the data collection procedure) can be repeated with the same results (Yin 1994 p.33).

Validity can be tested on three dimensions:

• Construct validity refers to the establishing of correct operational measures of the

concepts being studied.

• Internal validity refers to the establishing of a causal relationship, whereby certain

conditions are shown to lead to other conditions, as distinguished from spurious relationships. Internal validity is only applicable for explanatory or causal case studies, and not for descriptive or exploratory case studies.

• External validity refers to the establishing of the domain to which a study’s findings

can be generalized (Yin 1994 p.33).

A research protocol contains the research instrument(s), the rules and general procedures for using the instrument(s), an indication of the sources of information, the set of questions to be used in interviews and a guide for case study report (Yin 1994). In the remaining of this section we show how we elaborated our research protocol.

Action plan

The first step in conducting the empirical research was the development of an action plan. This action plan was specified per individual case and consisted of an agreement between the contact person of the company and the researcher about the time period of the research, the research instruments used, the respondents and the expected results of the research for the company. Table 5.2 presents the general agreements made.

Table 5.2 Agreements with the companies studied (by means of an action plan)

Subject Agreement

Time period

Two weeks per company.

Interviews with managers: about 2 hours;

Interviews with employees involved: about 4 hours; Interviews with employees indirectly involved: about 1 hour. Research

instruments

Use of interviews, accessibility of relevant documents, opportunity of accompanying employees during the processing of orders, opportunity to attend relevant meetings, guided tour through the production plant.

Respondents

Manager Sales, Manager Operations, Managing Director.

Employees involved in order processing: mostly employee(s) sales desk, planner(s)

Employees indirectly involved: Design, Engineering, Logistics, Quality Department, Information Systems.

Results For the company: Diagnosis of the ordering process. For the researcher: Data gathering for empirical research.

Research instruments

During our research we used multiple sources of evidence by using various research instruments, necessary in obtaining construct validity. The instruments used are:

• Studying relevant documents, such as management reports, order book, minutes of

meetings, quality handbook and work instructions. The documents were used to corroborate and augment evidence from other sources (Yin 1994).

• Observing the ordering process and all occurring activities by means of

accompanying the employees involved during the processing of the order. The observational evidence provided additional information about the issues involved in order processing.

• Attending pre-arranged and spontaneous meetings concerning the processing of

orders in order to get a better understanding of and additional information about the issues in order processing.

• Interviewing different actors involved in the ordering process:

- Semi-structured interviews with employees directly involved. - Semi-structured interviews with involved managers.

- Open interviews with employees indirectly involved.

By interviewing multiple respondents per case we applied triangulation in our research. The set of questions

The interviews with multiple respondents were central to the data collection. We chose for using semi-structured interviews realizing a comparable basic set of questions and, at the same time, having the opportunity to further explore relevant issues coming up from interviews with individual respondents. The questions were linked with the operationalizations of the variables to be studied. Four sets of interview questions were developed:

• Questions for managers Sales (MS), see appendix I, • Questions for managers Production (MP), see appendix II

• Questions for employees of the sales function (S), see appendix III

• Questions for the employees of the planning and /or production function (P), see

appendix IV.

These sets of questions, directly linked with operational measures, served as a basis for developing a chain of evidence (Yin 1994).

Procedure for using the instruments plus sources of information

As part of the research protocol we made a procedure for using the instruments and sources of information. We made an overview in which we defined per variable: the operational measure(s), the research instrument(s), the source of information (an indication of the actor or the document where the information should be sought) and the accompanying interview question. An example is given in Table 5.3.

Table 5.3 Example of elaboration per variable for research protocol

Variable Operational measure Instrument Where/ who Question

Heterogeneity demand # PMC’s or product groups # customers # regular customers # incidental customers # classification of customer # average order lines per order

Documents Interviews Sales budget/ sales forecast Managers Sales Planning MS3, MP3 MS13, MP18 S11,S23, S36 P10

Guide for case study report: case study database

On the basis of our overview of operational measures we developed a code list for coding the data gathered. The interviews were transcribed – shortly after the interview took place - and translated into the software program ATLAS/ti. Then, the interview reports were coded on the basis of the code list. Thus, we developed a case study database. In this case study database not only the coded interviews were reported, but also information from observations and relevant documents. The database provided an overview of the data gathered per individual case and guided the case study reports for the separate companies studied. The case study database was therefore an important tactic in realizing reliability of the empirical research.

Actor Activity Diagram

A central element of the case study report per company was the description of the ordering process. In describing the ordering process per company we used Actor Activity Diagramming (AAD) as an important modeling technique. AAD is a modeling technique to visualize business processes in terms of activities, actors and transitions (transfers) between actors (Schaap, 2001). AAD can be traced back to Role Activity Diagramming (Ould 1995) used in the field of Business Process Reengineering. It is different from Role Activity Diagramming by using the concept ‘actor’ in stead of the more abstract concept ‘role’ and by using a more limited set of symbols and drawing rules (Schaap, 2004). As a result, AAD is an appropriate instrument in modelling the order flow in terms of actors, activities and transitions.

An Actor Activity Diagram (AAD) shows in one glance what actors are involved, what activities take place and in what sequence (=the workflow). It further shows clearly at what moments during the ordering process information is handed over to another actor or consultation must take place before an order can be further processed. Thus, AAD is specifically suited to model the information processing dimension (information flow; information processing activities and information capabilities used) and the organizational setting of the ordering process (actors involved, information transfer and coordination). The use of AAD is less appropriate to model the logistical decision-making dimension, as it is not possible to model decisions to be taken and various levels of control. However, the diagram offers the possibility to study, for

instance, if order-processing activities in which logistical decisions are involved are supported by information systems and what actors are involved.

Figure 5.1 shows an example of an AAD. The actors involved are shown above the columns. On the right we added a column ‘information system’ to show what kinds of information systems are used during the ordering process. The transformation of the customer order into order agreements and a production order is shown vertically. Transactions or handing over of information is shown horizontally. For an explanation of the symbols see the legend in Appendix V.

C u stom er E m p loyee _{S a les d esk} P la n n er U n it lea d er _{P rod u ctio n} In fo rm a tio n _{S ystem}

M ak e ord er C h eck ord er d ata d elive ry tim e P la n ord er E n ter o rd er d ata o rde r E R P -system E R P -system S p rea d sh eet P la n nin g c on fir- m a tio n P rod . o rde r

Review of case study report

For the companies involved in the research a case study report was written in which the main focus was on diagnosing the ordering process of the company and to report on strengths and weaknesses.

The managers involved per company reviewed the reports. After review we planned a separate session per company to discuss the results of the study. During these sessions at least the manager sales and the manager production were present and in most companies also the managing director. That way, verification of the data gathered and of the analysis of the data was guaranteed from the perspectives of the various functional disciplines involved in order processing. Furthermore, not only the researcher but also an independent researcher was present during the sessions to monitor validity.

The discussion during these sessions confirmed the main findings of the study and also showed the importance of the AAD as diagnosis instrument for the managers.