[KH10] and Kruschitz [Kru09].
An alternative XML encoding for design patterns is that of EML. First introduced in Welicki et al. [WLA05], EML is an XML schema for describing all kinds of patterns and supporting concepts. The work by Welicki et al. [Wel+06] continued in Welicki et al. [WLA06; Wel+06]. Rather than describe textual documents, EML looked to the encoding of the pattern itself (software artefacts) in the XML schema. This is a marked change from the previously discussed schema.
In a similar attempt to Welicki et al. [WLA06], Mana et al. [Man+13] looked to describing security patterns within XML schema. Unlike Welicki et al. the work of Mana et al. concentrated on the encoding of security requirements, software artefacts in the form of UML, and the relations between the two. Further, Mana et al. provided more textual documentation into their schema in comparison to the work of Welicki et al.
Regardless of the presented XML encoding, a trade-off exists when encoding design patterns as an XML schema between: encoding the human readable document; encoding the pattern artefact; and encoding the link between different patterns.
2.7
Pattern Evaluation
Unfortunately, not all patterns created are in fact patterns [WC02; Hey+07], and pattern evaluation is one of the lesser reported aspects within pattern research. When looking to evaluate a pattern one must determine whether the problem is satisfied by the solution, how ‘tried and testing’ the solution is, and how well documented the pattern is. Further, the evaluation of a pattern must be reproducible, consistent, and allow for fine- grained analysis of the presented pattern. There are several known evaluation practises used within the pattern community, namely:Peer Review,Shepherding[WF11], and Writer’s Workshops. However, these evaluation systems do not provide comprehensive
guarantees inallthese areas. Nor do some evaluation systems provide comprehensive
guarantees that the method is reproducible, consistent, nor allow for fine-grained analysis.
One of the difficulties in constructing an evaluation system for software design patterns is that the subject domain covered is heterogeneous. A single general purpose evaluation system cannot be, and should not be specified. For example, Bunke et al.
2. Design Pattern Engineering
[BKS11] detail that, among other things, not all patterns adhere to common pattern templates. Any suggested evaluation process must be tailored to the patterns being evaluated. The remaining part of this section on evaluation discusses several known evaluation techniques used.
2.7.1
Shepherding
Shepherding is the recommended pattern evaluation technique for the PLoP series of conferences: PLoP, EuroPLoP, and Viking PLoP. Shepherding is a process in which experienced pattern writers (shepherds) are paired with pattern authors—theirsheep.
Harrison [Har99] details a common shepherding process, and provides implementa- tion guidance. The goal of shepherding is to capitalise upon the experienced writer’s knowledge in creating patterns to guide and provide expert commentary on the presen- ted pattern.
However, the shepherding process is a generalised technique that is applicable to all patterns. It does not provide fine-grained nor reproducible guidance over how to measure what constitutes a good pattern from a bad pattern. Nor does the process detail how the solution presented should be evaluated to determine its quality. Further it makes a tacit assumption that the advice of Meszaros and Doble [MD97] was used to drive the writing of the pattern being evaluated. The advice offered by Meszaros and Doble [MD97] tacitly provides a means to ensure good quality patterns by guiding the authors to writing patterns in a good style. Other existing pattern writing guides such as Wellhausen and Fießer [WF11] such Harrison [Har04] also detail how notions of quality are accounted for during. The Shepherding process provides subjective evaluation over the quality of the pattern itself and not the solution being described.
2.7.2
Writers Workshops
Writer’s Workshops are moderated Socratic discussions involving sets of like patterns. Involved in these workshops are the pattern authors themselves, and a set of peers. Traditionally, during the workshop the pattern author is asked to sit aside from the group and listen to the discussion of their pattern. The group moderator leads the discussions, ensuring that the strengths and weaknesses of the pattern are discussed and that suggestions for improvements are made and noted. Schmidt [Sch06], Coplien 22
2.7. Pattern Evaluation and Woolf [CW97] and Gabriel [Gab02] presents commonly used guidance used by
the pattern community for such workshops.
Remark. AtPLoP 2015a new style of writer’s workshop was trialled in which the
pattern author had more involvement in the discussion. This involvement was to spe- cifically introduce the piece of work, provide a set aims for the feedback, and provide clarification of points raised by participants.
2.7.3
Structured Approaches
Both Shepherding and Writer’s Workshops provide informal approaches to pattern evaluation. There are, however, more structured approaches.
Heyman et al. [Hey+07] presents two methods of evaluation: the first determ- ines if the presented pattern is a pattern; and the second provides an assessment over documentation quality. For the former, Heyman et al. uses a set of simple criteria to investigate thepatternessof the presented pattern. Quality of documentation is
assessed using simple qualitative metrics to asses content quality per expected heading. With each heading being weighted, a final score can be given to determine the overall adherence to documentation quality. However, this approach concentrates of pattern presentation and does not evaluate the pattern in other areas. For example, goodness of the solution to address the problem.
Halkidis et al. [HCS04] performed a qualitative analysis of several security patterns according to: (a) how well the pattern adheres to ten guiding principles for building secure software [VM11]; (b) how well the pattern deters the software developer from building an insecure system; and (c) how the pattern responds to different types of attack. Similar work was also performed by B. H. Cheng et al. [Che+03]. The use of qualitative evaluation criteria as used by Halkidis et al. [HCS04] is inherently prob- lematic. Halkidis et al. [HCS04, Section 4] mention that some of the criteria specified can only be used to assess the patterns implementation and not the pattern itself. For a qualitative analysis of patterns, criteria assessing the patterns and not implementation needs to be specified. The approach by Halkidis et al. is purely for security patterns and concentrates of assessment of the quality of solution and not quality of documentation.
Laverdière et al. [Lav+06] presents a comprehensive set of criteria for security design pattern evaluation using theSix Sigmaapproach. As with the approach taken by
Halkidis et al. [HCS04], this techniques concentrates on security patterns. However, the guidance established by Laverdière et al. [Lav+06] does provide a more holistic
2. Design Pattern Engineering
treatment towards design pattern evaluation. Unfortunately, no implementation guidance, nor results in using this technique were presented.
An underreported aspect of design pattern evaluation is that of usability. Thimthong et al. [TCK13] explore the use of user studies for pattern evaluation. However, use of user studies should be limited as the usefulness of such studies can be ineffective and unhelpful if done improperly [GB08].
DuringPLoP ’15, Xia et al. [Xia+15] introduced a more structured approach to pat-
tern evaluation to bolster writer’s workshops. The authors took established evaluation methodologies (checklists and code review) from software engineering and applied them to evaluate design patterns. Checklists are used to represent guiding evaluation criteria for design patterns, and detail what reviewers should look for. Perspectives are used to guide the review from the view-point of an entity involved in the pattern construction and domain of operation.