3.8
Qualitative Evaluation
Among the main constituents of the proposed approach, only our incremental layout opti- misation technique has been evaluated on a representative application. A practical assess- ment of the combination of our contributions against a significant case study would have been desirable to confirm their effectiveness. In this respect, an ideal evaluation would have consisted in reengineering a representative part of a real industrial application fol- lowing our development approach, and then compare the resulting application against the original system with respect to both analysis cost and quality of the results. Unfortunately, the prototyping state of the tools we selected to support our investigation prevented us from realising a sufficiently complex software system. We are confident that we will soon be able to use a more mature release of the SCM modeling tool, which should allow us to carry on an extensive evaluation of our approach.
This notwithstanding, it is still possible to evaluate our contribution against its indus- trial applicability. When defining our approach, in fact, we claimed that it lends itself to an effective application to the industrial development process. More importantly, we also contended that the proposed techniques facilitate the application of timing analysis techniques in the industrial development. To evaluate our approach from a qualitative (as opposed to quantitative) standpoint, we leverage on the high-level industrial issues on the applicability of timing analysis, that have been introduced in Section 3.3.2, as evaluation criteria to assess whether our approach contributes to narrowing the gap between state-of- the-art timing analysis and HIRTS practice. Table 3.5 evaluates our approach with respect to the identified industrial issues.
Scalability. Although our approach does not explicitly address the scalability problem,
the MDE framework we propose may help in cutting the complexity incurred by inter- procedural analysis and context-dependency, which are the root causes of the (software- related) state space explosion. Both the architectural and functional specifications con- tribute to the overall complexity of the software and can be tuned to disallow ill-formed software structures (e.g., huge call-graphs) or adverse constructs (e.g., unstructured multi- way branches).
Required skills and knowledge. As we observed in Section 3.3.2, the provision of trust-
worthy and precise annotations to guide the analysis process typically ask for specific skills and deep knowledge of the program under analysis. The generation of better analysable
Id Issue Contribution
I 1 Scalability Indirectly reduced as a side effect of the proposed MDE framework
I 2 Required skills and knowledge Mitigated by the enforcement of better analysable software
I 3 Relationship with schedulability analysis Addressed in the functional specifica- tion of a system
I 4 Perceived quality of the results Reduced pessimism by construction and by the extraction of precise flow- fact information directly from the soft- ware model
I 5 Cost-efficiency Reduced impact of the time-consuming definition of flow facts
I 6 Extensive tool support Addressed with respect to the proposed layout optimisation technique
I 7 Integration in the SW life cycle Improved support to incremental devel- opment
Table 3.5: Evaluation of our approach against the industrial requirements.
software and the automatic extraction of flow facts both contribute to reducing the amount of user intervention and relieve the user from reconstructing the information on the pro- gram. No particular skills are required, besides those generally assumed for a software designer.
Relationship with schedulability analysis. Inattentive system-level design choices may
complicate the separation of intra- and inter-task timing analysis. The factorisation of timing-aware design choices, as those identified in Section 3.5.2, in the architectural spec- ification of a system facilitates a clear separation between system-level and task-level con- cerns.
Perceived quality of the results. The application of our approach may help improve
the quality of the results of timing analysis in two respects. Firstly, the avoidance of poorly predictable code constructs is expected to reduce the level of pessimism incurred by timing analysis. Secondly, the automatic extraction of flow-fact information from the model guarantees a high level of precision in the applied annotations.
3.8 Qualitative Evaluation 123
Cost-efficiency. Again, the impact of the time-consuming task of collecting and defining
flow-fact annotations in the overall costs incurred by timing analysis is explicitly addressed and mitigated by our approach through the enforcement of more analysable code and the automatic generation of a large part of the required annotations.
Extensive tool support. The quality of the support provided by current timing analysis
tools is out of question. Our approach instead explicitly addresses the need of automated support for memory layout optimisations. In Section 3.7 we provide a fully automated prototype tool for the computation and enforcement of our incremental layout optimisation approach.
Integration in the SW life cycle. On the one hand, our contribution relies on the real-
isation and adoption of a non-standard MDE framework. Although the MDE paradigm is increasingly adopted in HIRTS, the strong orientation of our approach towards timing analysis concerns does not allow an effortless integration in a consolidated industrial tool- chain. However, evaluating the distance of our approach from a generic MDE framework against the benefits that can be obtained may justify an investment that would be applied once for all.
On the other hand, our layout optimisation approach, besides reducing the cache- induced variability, explicitly accounts for incrementality as a main characterising trait of the industrial development and facilitates an earlier application of WCET analysis on incremental releases, instead of the final system.
Chapter 4
Conclusions and Future Work
4.1
Recapitulation of study objectives
The need for more computational power to meet increasingly complex user demands is driving even the most conservative high integrity real-time systems industry towards the adoption of more complex processor equipped with caches and other acceleration features. The introduction of caches, in particular, induces a highly variable timing behaviour that do complicates both schedulability and timing analysis.
The industrial stakeholders understand the fact that the migration to cache-equipped processors is likely to hike the effort required in analysing the timing behaviour of a sys- tem, to the extent of breaking an already delicate balance between time and cost of timing analysis and the quality of its results. On the one hand, in fact, the current industrial ap- proach to timing analysis, often still based on simulation and testing, is definitely poorly equipped to cope with the variability incurred by caches. On the other hand, the application of advanced WCET analysis techniques on large-scale complex industrial software devel- oped without analysability in mind, hits on the inherent limitations of those approaches.
In our thesis, we contend that the disruptive effect of caches in the qualification of industrial-level HIRTS can be effectively and efficiently governed only by imposing a paradigm shift in the industrial practice towards an informed use of caches and a proactive approach towards timing analysis. In particular, we propose the adoption of a structured "cache-aware" approach aimed at (i) allowing a cost-effective application of state-of-the- art WCET analysis to complex systems; and (ii) minimising the variability and unpre- dictability incurred by caches. We maintain, in fact, that any countermeasure to the cache- induced variability is unavoidably tied to a more rigorous attitude towards timing analysis,
more commensurate with the industrial requirements on timing predictability.
We intend such cache-aware approach as the structured combination of a set of coun- termeasures to improve cache predictability and, in a broader sense, to facilitate the appli- cation of timing analysis in industrial setting. As fundamental requirement, the identified techniques and methods shall allow an efficient integration and application in the HIRTS industrial development process.