In order to prove that the SIN LP and SIN RP algorithms (cf. Sections 5.5.1 and 5.5.2) actually support process participants involved in knowledge-intensive business processes, we applied the algorithms to a real-world use case from the automotive domain (cf. Section 9.2.1). Specifically, we implemented the algorithms (cf. Section 9.2.2) and then compared their outcome with the results of a survey among experienced automotive engineers. The latter were asked to manually rate the relevance of process information related to the considered use case based on their own experiences (cf. Section 9.2.3). Results indicate that the algorithms can indeed replace the costly and time-intensive human determination of relevant process information (cf. Section 9.2.4).
In particular, the survey has been guided by two research questions (cf. Table 9.1):
# Research Questions
RQ1.1 How do the results of the SIN LP algorithm match with user- generated evaluations on the relevance of process information?
RQ1.2 How good is the ranking of process information based on the SIN RP algorithm compared to other ranking approaches?
Table 9.1: Research questions underlying the survey.
9.2.1 Use Case
The considered use case deals with the review of product requirements as documented in functional specifications at a large automotive OEM. The goal is to improve as well as to approve such specifications. The corresponding review process is knowledge-intensive since it requires large amounts of process information (e.g., protocols, checklists, guide- lines, manuals, and review results), user interactions (e.g., “perform review meeting”, “send review comments”), and decision-making (e.g., shall the document be approved or not?). Three roles are involved: (1) the author provides the specification to be reviewed, (2) the review moderator organizes the review meetings, and (3) the reviewer analyzes the provided specification and records errors, ambiguities and uncertainties.
The review process (cf. Figure 9.2) starts with the preparation of the document to be reviewed (Task T1). This task is performed by the author of the document. Based on this initial preparation, the author decides whether or not a preliminary review meeting becomes necessary (Task T2). Afterwards, the document may be reviewed (Task T3). Based on the outcome of the review, the reviewer decides whether an additional review meeting is needed (Task T5) or whether it is sufficient to directly send findings and comments to the author (Task T4). The latter then evaluates review results (Task T6) and updates the document accordingly (Task T7). If the overall review status of the document is “rejected”, it will not be approved. In turn, if the overall review status is “accepted”, the author may finally approve the document (Task T8). For each of these tasks, a variety of process information is needed, e.g., guidelines and templates.
prepare document for review (T1) x perform preliminary meeting (T2) x perform review (T3) x send review comments (T4) evaluate review com- ments (T6) perform review meeting (T5) x preparereview results (T7) x check review results (T8) Doc. draft Doc. ready Feedback Reviewer Review Protocol Review Process Author Moderator Reviewer Doc. ready Review Protocol Doc. reviewed Review Results Review Protocol Findings Review Results Findings Findings Findings Review Protocol should a preliminary meeting be held? which review procedure?
should the document be approved or not?
Doc. ready
Figure 9.2: Use case: review of product requirements.
9.2.2 Implementation
Based on the discussed use case, we first implemented the corresponding SIN. Altogether, it comprised a process schema, three process instances, and about 300 documents (i.e., process information) such as review protocols, guidelines, and review results. For cre- ating the SIN, we used the semantic middleware iQser GIN Server as well as several Java open-source plugins we had developed in this context. The implemented SIN in- cludes 348 objects (45 process objects, 303 information objects) and 65,991 relationships (77 process object relationships, 65,319 information object relationships, and 595 cross- object relationships). While Figure 9.3 shows the entire SIN of the use case, Figure 9.4 only depicts objects (i.e., information and process objects) directly related to Task T3. For privacy reasons, the document names have been blurred in the following screenshots. We then implemented the algorithms in a prototype called iGraph (cf. Section 8.2), a web-based Java application. iGraph uses the web framework Play 2.1.1, the web engine
Figure 9.3: Entire SIN of the use case.
Bootstrap 2.3.1, the JavaScript library jQuery 1.8.3, the database MySQL 5, the text search engine library Lucene 2.4, the JavaScript library D3 3.1.1, HTML5, and CSS3.
9.2.3 Empirical Validation
In order to validate the SIN LP and SIN RP algorithms, we conducted a survey in the automotive domain. In this survey, automotive engineers evaluated previously calculated results of the two algorithms. With this survey, we want to show that the algorithmic results can indeed replace the costly and time-intensive human determination of relevant process information. More specifically, the goal is to prove the accuracy of the SIN LP and SIN RP algorithms.
RQ1.1 (Investigating the SIN LP algorithm). To investigate RQ1.1, we used iGraph to calculate two link popularity result lists. As input values, we set init = 0.45, i = 12, and d = 0.5. Moreover, we double weighted “is similar to” relationships since these relationships are usually more important than others in a SIN (cf. Section 5.3.1).
Overall, we received two result lists: The first list constituted the top eight documents according to the SIN LP algorithm for Task T1 (“prepare document for review”). In turn, the second list constituted the top eight documents according to the SIN LP algorithm for Task T3 (“perform review”). Table 9.2 shows the documents (i.e., process information) the SIN LP algorithm returns for Tasks T1 and T3. We then asked survey participants to evaluate the relevance of the calculated documents for both process tasks.
# ID Type SIN LP Marked Ratio Relevant?
T1 1231 Review Template 0.443 12 60.0 % X 1210 Process Overview 0.442 20 100.0 % X 439 Review Template 0.441 4 20.0 % 432 Specific Review 0.439 17 85.0 % X 811 Guideline 0.435 4 20.0 % 439 Protocol 0.434 2 10.0 % 578 Checklist 0.434 19 95.0 % X 777 Guideline 0.432 19 95.0 % X T3 1210 Process Overview 0.443 17 85.0 % X 879 Protocol 0.442 19 95.0 % X 431 Specific Review 0.441 10 50.0 % 432 Specific Review 0.439 9 45.0 % 741 Review Template 0.435 7 35.0 % 439 Review Template 0.434 6 30.0 % 578 Checklist 0.434 18 90.0 % X 729 Review Template 0.432 19 95.0 % X
Table 9.2: SIN LP algorithm validation results.
As can be seen from Table 9.2, the survey participants confirmed the relevance for the majority of the 16 documents identified by the SIN LP algorithm. Note that we
consider a document as being relevant if more than half of the survey participants confirm relevance. Moreover, the results show that the SIN LP algorithm is indeed well working, especially since its overall accuracy can be further improved, for example, by combining it with other algorithms (e.g., the SIN RP algorithm).
RQ1.2 (Investigating the SIN RP algorithm). To investigate RQ1.2, we first calcu- lated a ranking of review templates with the SIN RP algorithm based on real-world ratings we obtained from the automotive OEM supporting the survey. Additionally, we calculated three rate-based rankings on Formula 5.8 (ranking based on the total number of ratings), Formula 5.9 (ranking based on the average rating), and a random rank- ing. For example, Figure 9.5 shows a ranking of PDF documents (mainly guidelines) according to the SIN RP algorithm.
Figure 9.5: Ranking of PDF documents according to the SIN RP algorithm.
We then asked survey participants to evaluate both the plausibility and the usefulness of the four rankings. Figure 9.6A shows that 16 out of 20 participants consider the ranking created with the SIN RP algorithm as the most plausible one. The ranking based on the total number of ratings is considered as the second most plausible one with three votes. The ranking based on the average rating only received one vote by the participants. The random ranking received no votes. Moreover, as aforementioned, we asked the participants to evaluate the usefulness of the rankings based on a 5-Likert scale [284] ranging from “not at all useful” to “very useful”. Figure 9.6B shows that 87.5% of the participants stated that the ranking created with the SIN RP algorithm is
“useful” or “very useful”. Again, survey results confirm that the SIN RP algorithm is indeed performing well and can support participants during daily work.
1 2 3 4 5 1
Question: Is the document ranking of
the SIN RP algorithm useful? a
1: not at all useful, 2: not very usefull, 3: neutral, 4: somewhat useful, 5: very useful
lower quartile minimum median maximum upper quartile 0 16 1 3 0 5 10 15 20 (d) (c) (b) (a)
Question: Which is the most plausible
document ranking?
(a) TotalNumber algorithm, (b) AverageRate algorithm, (c) SIN RP algorithm, (d) Random
A B
Figure 9.6: SIN RP algorithm validation results.
9.2.4 Conclusion
The considered automotive scenario confirms that most of the documents returned by the SIN LP algorithm are indeed relevant (RQ1.1). Moreover, the empirical research shows that the link popularity constitutes a good indicator for identifying relevant process information, especially since results of the SIN LP algorithm can be further refined for specific tasks by applying the SIN LP algorithm to only specific parts of the SIN (e.g., to a particular process task, corresponding task instances, or related information objects). The results of the SIN RP algorithm are considered as being very useful by the partic- ipants as well (RQ1.2). In fact, most participants state that the ranking of documents as suggested by the SIN RP algorithm is both plausible and useful. Additionally, the algorithm avoids the problematic situation that process information with only few good user ratings is directly ranked on the first position. Finally, note that the results of the algorithm can be further improved, for example, by taking the expertise of users into account; i.e., ratings of experienced users might be weighted higher.
In summary, the popularity values of the algorithms (cf. Sections 5.5.1 and 5.5.2) clearly help to determine the relevance of process information. However, as it is difficult to determine the overall relevance of process information based on a single algorithm, we will combine the algorithms when extending the POIL framework at a later stage.