Evaluation Procedure

Following the suggestion described above, primarily by Kaplan and Duchon (1988) and Thomas (2006), the underlying evaluation was processed in 5 steps, illustrated in Figure 25.

6 Evaluation 88

(1) Upfront quantitative and qualitative data were collected from each individual team member. For this step, a brief pre-questionnaire was applied with open questions, allowing for a more individualized conducting of the interviews20. The pre- questionnaires were submitted to the designated team members and collected by the researchers a few days before the on-site evaluations. The pre-questionnaire was divided into two parts. The first of these covered open questions to characterize the participant current position in the context of supply management, while the second covered current supply management problems in the company overall and with the software currently in use. The second part covered six statements concerning the required scope of supply management software, applying Likert scales from 1 to 5.

(2) Starting the actual evaluations at the participants’ companies, the participants as a whole were familiarized with the idea of supply networks and the terminological foundations used to define and describe the problem setting underlying the project. This part was based on the insights gained during phase 1 of this research.

(3) Each team was split up into sub-teams, each comprising three roles: a supply management professional, a sales professional and an information management professional in IT. Together, they executed a quote-to-invoice use case with the deployed artifact B-Zone. All roles where covered by supply management professionals from the respective company, whose current position in the company comprised a prime focus on procurement and information management. Each of these sub-teams was introduced to the artifact concept version and asked to perform a quote-to-invoice process using the artifact. The supply professionals were asked to perform their respective roles, while the information management professional was asked to observe and assess the artifact’s characteristics.

During the task of the quote-to-invoice use case, the supply management participant was instructed to imagine him or herself as a category manager from the purchasing perspective, being responsible for a certain set of supplies, specifically IT equipment. The sales representative was asked to put him or herself in the position of a category manager from the sales side, being responsible for sales and marketing of defined product categories, in particular IT hardware. According to the use case descriptions provided to the participants, the purchaser is searching for a specific IT hardware supplier who s/he met during a conference. The purchaser finds the contact using the search capabilities and digests the related business performance information of the potential supplier, available in B-Zone. By selecting one of the pre-defined business templates, s/he creates a quotation request for a specific laptop offered by the supplier. The supplier receives the quote in his or her news feed but cannot deliver the complete quantity at the requested delivery date. Consequently, the supplier posts a quote with an update on the later delivery date, granting a rebate due to the late delivery. The purchaser receives this notification via the news feed in B-Zone and selects in the business zone area of the artifact a more specific business template for orders, including schedule lines. The purchaser splits up the quantity between the requested delivery data

20

order, confirms it and triggers the delivery and the invoice. All the status updates on business objects from the supplier site are made transparent in the news feed for the purchaser, who finally confirms the delivery, and reviews and approves the invoice for payment.

(4) Once the scenario had been completed, each professional was interviewed individually. These interviews were semi-structured using an interview guideline, with an average duration of 40 minutes21. All the interviews were recorded and transliterated verbatim. Completed transcripts were submitted to the informants for correction, adjustment, and approval22. All field settings were performed with two of the researchers as facilitators and observers present on site. These also gathered observations and impressions that were used to discuss the emerging results with the entire DSR team.

The rationale for interviewing ever participant from the teams, in addition to their participation in the scenarios tests in attendance of the researcher, was to allow for data and investigator triangulation (Mayring 2001; Flick 2009; Flick 2010) - using researcher’s observations, the participant’s experiences, and their performance results. This creates considerable potential to “[…] readily assess the completeness and plausibility of the participant’s account, thus making inconsistencies and contradictions more visible” (Schultze and Avital 2011, p. 5). It is believed that this is an important contribution to the study’s validity and reliability.

(5) After completion of the field studies, all qualitative material gathered was coded, analyzed, and theoretically abstracted following established methodological principles (Corbin and Strauss 1990, Myers 2010, Strauss and Corbin 1990, Thomas 2006). Atlas.ti 6, a software tool specialized on qualitative data analysis, coding and categorizations, supported the respective data coding and analysis process. Firstly, a code was allocated to every important part of the interview. The code list was then checked for duplicates or similar entries, which were merged into a single code. Finally, the remaining codes were synthesized into core categories which describe the interview’s main issues.

The insights gained from these field studies were then used to assess how well the artifact performed in the live scenario, how well the design principles employed actually match the problems relevant to practitioners, and what additional design choices would be needed to build a more useful and better performing artifact.