DSL evaluation

DSL users need to know the inherent strengths and domain applicability of the languages they use [209]. Since DSLs fall under the software language umbrella, applicable soft- ware language evaluation methods can be used for DSL evaluation. Evaluation criteria of software languages can be classified into four categories as follows: i ) Application domain; ii ) Human factors; iii ) Software engineering; iv ) Language design and imple- mentation [209].

Application domain evaluation concerns, also known as the philosophy of design [210], entail such criteria that assess how well a language supports developing software for a specific type of applications [211, 212]. Incompatible abstractions of the problem domain between software language users and software language engineers are a constant challenge to language usability [213]. Software language users need to know the inherent strengths and domain applicability of the languages they use, in order to use the software languages effectively [209].

Human factors come into play noting that software engineering is human-intensive, as well as that such criteria can be used to assess the user-friendliness of a particular DSL [214]. Factors such as naturalness and readability of software languages have not been consistently endorsed by evidences in software language engineering [215]. DSLs can be regarded as user interfaces as they provide a domain-expert with an interface to a desired computation platform [216]. Like other software languages, DSLs have rarely received usability or human factors evaluation which may have led to the deployment of inadequate and unusable languages [217].

Software engineering concerns are those aspects that enable the development of good software by evaluating a software language’s capacity to support such qualities as relia- bility, portability, re-usability and maintainability [209,218]. A number of researchers have argued that DSLs can improve desirable attributes of resulting software [182,219].

Language design and implementation concerns are those aspects that assess how well a language is designed to enable support of modelling tools such as editors, compilers and interpreters [220]. Modelling tools for a particular DSL are key to the DSL’s acceptance in its modelling community [221,222]

Several studies have investigated quality characteristics of DSLs that can guide DSL eval- uations. Though there have been some studies on DSL assessment, there is currently no agreed standard for DSL evaluation [223]. Kolovos et al. [224] argue that characteristics that are used for evaluating general-purpose programming languages can also be ap- plied when evaluating DSLs though their relative importance may differ. Kahlaoui et al. [225] propose an approach that identifies a set of success factors that be transformed into a project-specific set of assessment criteria for DSL evaluation. K¨arn¨a et al. [226] propose a DSL evaluation approach that evaluates a DSL against pre-set goals through user experiments. Mohagheghi and Haugen [227] also propose a set of DSL assessment criteria based on their experiences in practice. Haugen and Mohagheghi [228] propose a multi-dimensional framework for characterising DSLs using a questionnaire across three dimensions: expressiveness, formalization and transparency. Wu et al. [229] propose a slightly different approach that can determine the level of effort required in developing and using a DSL through metrics that can measure the effort. Karsai et al. [180] propose a set of DSL design guidelines that can be considered when developing a DSL and classified the guidelines using development phases as follows: i ) Language Purpose – guidelines for the early activities of the DSL development process; ii ) Language Realization – guide- lines which discuss DSL implementation methods; iii ) Language Content – guidelines which focus on characteristics of language elements; iv ) Concrete Syntax – guidelines that contribute to readability of a language; v ) Abstract Syntax – guidelines that affect the internal representation of a language. Frank [1] proposes generic requirements of a DSL as follows: i ) Concepts of a DSL should correspond to concepts that are familiar to prospective DSL users; ii ) A DSL should provide domain-specific concepts so long as their semantics are invariant within the scope of the DSL’s application; iii ) A DSL’s concepts should allow for modelling that is sufficient for all foreseeable applications; iv ) A DSL should provide concepts that can clearly distinguish different abstraction levels within a model; v ) A clear mapping should exist between a DSL’s concepts to the concepts of the relevant target representation. DSL developers can evaluate a particular DSL against specific requirements that are derived from the generic requirements [1]. Kahraman and Bilgen [230] propose a qualitative framework for assessing DSLs that guides the selection of key DSL quality characteristics to guide an evaluator. Bariˇsic et al. [216] propose an experimental evaluation model for DSLs that focuses on usability and is based on techniques for the experimental evaluation of user interfaces. These previous studies highlight domain-appropriateness, simplicity, usability, formality and

expressiveness as important criteria that should be considered when creating a DSL to ensure that the resulting DSL is of high quality and that it can increase its likelihood of being accepted by its users.

Perceived usability can be regarded as an important high-level construct of usability [231, 232]. The subjective measurements of usability and the objective measurements of efficiency and effectiveness provide a comprehensive construct of usability [233, 234]. The International Standards Organization (ISO) defines usability as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use [235, 236]. The next section discusses standardised perceived usability measurement tools.