Measurement is as essential part of software engineering, as discussed by Pressman [278]. Park et al. [270] argue that software measurement allows the characterisation, evaluation, prediction or im- provement of the process. This concept of measurement is defined formally by the IEEE through a
metric[161], which is “a quantitative measure of the degree to which a system, component, or process possesses a given attribute.”
In the software engineering domain, these measurements may therefore be used to get some sense of whether the requirements are consistent, testable and implemented; whether the design is of high quality; to understand the complexity of the system; and whether the development activity can be improved [94, pg. 3–21]. Software metrics, in particular, can be combined in a well-defined way in order to aid analysis and assessment of a software system through asoftware quality model[94, pg. 338–344], such as the ISO 9126 model [166].
Individual metrics are defined for a particular domain, and often cannot be evaluated across dif- ferent domains. For example, web application metrics such as “number of pages” cannot be directly compared to implementation metrics as “lines of code” or “number of comments”. In this thesis, four domains of metrics will be discussed: domain-specific metrics for a particular web application (web application metrics); metrics for the modelling approach used (metamodelling metrics); and metrics for the development process itself (system metrics).
A number of other domain-specific metrics were considered, but have not been included in this thesis. For example, Selic [305] introduces the model-driven metrics ofcompilation timefor translat- ing a model, and theturnaround timefor applying incremental changes. Other metrics, such as code coverage [244, pg. 417] and execution time were also considered, however these metrics are more appropriate for highlighting deficiencies in a single implementation, and not for evaluating different implementations of the same system.
2.6.1 Web Application Metrics
Web application metricsare metrics evaluated against a particular implementation of a web applica- tion, and there is a significant body of existing research into defining and using these metrics. These metrics are often used to estimate the effort required to develop different applications, but they can also often be used to compare different web applications for complexity. They cannot be used to com- pare different implementations of thesamedesign, but are instead used to compare the complexity of different web application designs.
By reviewing a wide selection of research into web application metrics, Mendes et al. [224] found that most existing metrics are concerned with estimating the development effort for a given web ap- plication (cost estimation). As an example, Cowderoy [62] defines size metrics such as number of
13As discussed later in Section 8.3, Ticket 2.0has been implemented using the Symfony framework as Ticketsfand Ticketsf-mini, and using the modelling language proposed in this thesis asTicketiaml.
2.6 Metrics 27
Figure 2.4: The conceptual structure of theTicket 2.0benchmarking application, adapted from Wright and Dietrich [367]
Metric[235] [331] Description
TNP Number of packages, i.e.EPackages. NoC TNC Number of classes, i.e.EClasses.
NoAC Number of abstract classes, i.e.EClassesthat are specified asabstract. TNoR Total number of references, i.e.EReferences.
TNoA TNA Total number of attributes, i.e.EAttributes. NoD Number of primitive datatypes, i.e.EDataTypes. NoE Number of enumerations, i.e.EEnums.
Nav Navigability: The proportion of references with a definedeOpposite. Cont Containment: The proportion of containment references to all references. Dat Data quantity: The proportion of attributes to overall structural features.
Table 2.4: Selected metamodelling metrics for metamodels within the Eclipse Modeling Framework, adapted from Monperrus et al. [235] and Vépa et al. [331]
web pages; amount of text; number of images; number of features off-the-shelf; and include metrics dedicated to reuse or outsourcing.
Many metrics have been proposed for object-oriented software systems; for example, the MOOD, MOOSE, EMOOSE and QMOOD metrics are all different collections of object-oriented software metrics, as discussed by Baroni and Abreu [19]. Baroni [18] describes a comprehensive collection of software metrics, including these object-oriented metrics, that can be applied to a system. As a web application may be implemented using any conceptual methodology – including object-oriented, procedural and functional – these metrics cannot be consistently applied to web applications.
2.6.2 Metamodelling Metrics
Metamodelling metrics can be useful to compare different metamodelling approaches intending to model the same domain. These metrics can either be concerned with individual elements of the meta- model, or the metamodel as a whole. For example, Rossi and Brinkkemper [289] proposed a set of seventeen non-empirical – i.e., not derived from usability studies – complexity measurements for mod- elling languages, such as a metric for average complexity (C(MT)), and a metric for total conceptual
complexity (C(MT)). Siau and Cao [309] applied these metrics to the UML 1.4 specification [249],
arguing that these metrics illustrate that as a whole, the UML specification is very complex compared to other object-oriented modelling techniques.
Monperrus et al. [235] investigated applying similar metrics to modelling languages implemented in the Eclipse Modeling Framework14, and some of these proposed metrics are listed here in Table2.4. Vépa et al. [331] also look at a selection of metamodel metrics, in order to measure and compare different model repositories, and derive their metrics through a series of ATL transformations15; these metrics are also listed here in Table 2.4. The productivity of a modelling language could also be measured in terms of a function points per staff month metric; Mernik et al. [226] used this metric to measure the productivity of different programming languages.
In certain metamodelling implementations where metamodels are structurally represented as a graph16, one could also compare metamodels instances against standard graph metrics. For example,
14The Eclipse Modeling Framework (EMF) is described in further detail later in this thesis, in Section6.2.1. 15ATL transformations are described in further detail later in this thesis, in Section3.1.8.
2.6 Metrics 29
Metric Description
Tasks Tasks implemented
Time Development time, in whole weeks NDev Number of developers [305] FC Number of file changes Rev Number of SVN revisions
DTech Number of different programming technologies used [261] DMedia Number of different media types used [261]
Table 2.5: Overall system metrics, adapted from Selic [305] and Olsina et al. [261]
Metric Description
Files Number of files Size Size of all files, in bytes NCLOC Physical lines of code [269,94] ALOC Average NCLOC per file
Table 2.6: Language system metrics, adapted from Park [269]
thedegreeof a given class would refer to the number of attributes and references directly by the class, anddistinct typeswould refer to the distinct number of metamodel types used (e.g. classes, attributes, references, or enumerations). A suite of some metamodel graph metrics have already been proposed by Wright and Dietrich [369] to evaluate the complexity of EMF model instances.
2.6.3 System Metrics
There are a limited number of system metrics that can be used to compare two diverse implemen- tations of the same project, possibly because the development artefacts depend heavily on the staff and technologies selected for a given implementation. Metrics that are useful within one approach (for example, “number of plugins” in a Symfony project) may have no reciprocal metrics in another approach (for example, “number of visual models” in a MDD project).
In this research, it seems that only two common domains exist between different implementations of a given RIA.Overall system metrics may be used to evaluate the system size and development effort, as illustrated in Table2.5; a number of these metrics are derived from existing work by Olsina et al. [261] and Selic [305]. Similarly,language system metricsmay be used to evaluate the physical representations of the development artefacts, as discussed by Park [269].
With regards to both of these domains of metrics, frameworks allow three different types of de- velopment effort to be independently evaluated. As discussed earlier in Section2.2, these approaches can automatically generateboilerplate code, or encapsulate common functionality in runtime libraries. Consequently, system metrics should be applied to each of these different versions of a system:
• The manual effort, representing only content that has been developed independently of any boilerplate code or external libraries;
• The generated application, representing both the custom code and the generated boilerplate code provided by the framework; and
• Thecomplete application, representing all custom code, generated code, and included libraries. By splitting up system metrics into these three different versions, the level of complexity hidden or managed by the framework can be quantified. For example, external libraries can be expected to have a large number of files (Files); and the size of the generated application (Size,NCLOC17) should be significantly greater than the manual effort.