Summary

of them are rather complex to use as they are targeting the management of large software projects. That is why, they are not very popular among the HPC community where such data is often spread across multiple systems and performance experiments. Consequently, another software component was implemented as part of the middle layer of PAThWay in order to fill this gap and provide an integrated project documentation functionality. It is based on an customized version of the EclipseWiki editor for Eclipse and uses both Wiki-formatted text files and the internal database as back-end storage.

The bottom layer of the architecture of PAThWay consists of a set of supportive modules. These include a Git-based revision control system for tracking the code evolution during the typical performance tuning cycle, an environment detection tool for capturing the runtime configuration of experiments and an underlying execution and monitoring system based on the Parallel Tools Platform. In contrast to the previous software components, these modules are not directly visible to the users. Instead, they are only interfaced through the other components in order to build a more flexible and easily adaptable automation framework.

As part of the evaluation of PAThWay, this dissertation presented three popular performance en- gineering workflows that were successfully implemented within the framework. Firstly, a model for performing automatic scalability analyses of parallel applications was designed and then ex- tensively tested using the NAS Parallel Benchmarks Multi-Zone suite. Secondly, a performance engineering workflow for evaluating and comparing the runtime performance of memory sys- tems was discussed. In order to show its feasibility, the model was then executed on three dif- ferent HPC systems using the STREAM memory benchmark and the results were again shortly discussed. Thirdly, the standard benchmarking processes that is often applied in supercomputing centers throughout a pre-sales phase is introduced and subsequently evaluated using the well- established SPEC MPI-2007 benchmark suite.

What is more, this thesis proposed a complete BPMN model for semi-automatizing the very common performance tuning workflow. It discussed the importance of standardizing this process from the point of view of HPC centers and the positive effect that this would have on their daily operations. Additionally, a simple but effective methodology for building a collection of tuning recipes was introduced as part of the workflow. The ideas behind this workflow model aim at lay- ing the foundations for standardizing the way performance tuning is performed and at providing a real-world example on how HPC processes can be easily enhanced by adopting BPMN-based automation techniques.

Last but not least, this dissertation also discussed two new extensions to the Periscope perfor- mance engineering tool that were performed in order to create a more scalable, cross-platform analysis environment. Firstly, a portable memory analysis strategy was introduced and the moti-

vation behind its implementation was briefly discussed. Secondly, different statistical clustering techniques for summarizing large-scale performance data were presented and their suitability to the given problem was examined.

16.2. Future Work

This dissertation laid the foundation for process automation in the field of performance engineer- ing and proposed a prototype infrastructure for its realization using PAThWay. Consequently, these results also opened new research questions and motivated further implementation work. Among the many possible directions, three of the most important ones according to the author will be discussed in the following sections.

16.2.1. Interface with External Data Stores for Raw Performance Data

PAThWay provides a flexible framework for designing, executing and monitoring performance engineering workflows. Additionally, it is able to collect plenty of meta-information about the performed experiments such as runtime environment, execution logs and others. However, when it comes to organizing the raw performance data collected from these experiments, the framework has a very basic support. Even though it can store results from the Periscope toolkit directly into its database, data generated by other tools is neither processed nor internally stored. Instead, just a reference to the location of the collected information is saved within each experiment.

There are, however, other frameworks like TAUdb (previously called PerfDMF) [88] which are highly specialized in post-processing and management of large amounts of raw performance data. That is why, instead of extending PAThWay to perform such operations, these tools can be directly interfaced within the performance engineering workflows. For example, this way users will be able to perform cross-experiment comparisons or arbitrary manipulate the collected data in order to gain further insight about the runtime characteristics of their applications.

16.2.2. Automatic Post-Processing of Results and Process Reporting

Currently, only the configuration, execution and runtime monitoring of experiments is automated within PAThWay. The commonly following phase of results analysis is modeled as a manual BPMN activity which asks the user to explore the data and draw conclusions out of it on his own. This is, however, not a very productive approach and, to a certain extend, can also be easily automated. For example, after running all experiments in a scalability analysis workflow, the framework can directly process the collected results and extract key performance metrics out of it.

16.2. Future Work

Following, it can generate scalability diagrams or even create analysis reports out of the data on behalf of the user.

Furthermore, the framework can be extended to include an activity monitoring component which will be responsible for tracking and reporting the usage of PAThWay’s workflows. This information can be used within project reports or simply as a reference of the completed work.

16.2.3. Performance Engineering Standardization and New Workflow Models

Due to the early stage of PAThWay, only a very small set of performance engineering workflows were modeled using the framework. However, there are many other activities in the HPC field which can directly benefit from automation. That is why, to close up this list of future activities, the creation of further workflows is shortly discussed. It is believed that this will have the biggest impact on the future adoption of PAThWay as a workflow formalization tool and will trigger new discussions within community for the sake of standardizing the performance engineering field.

For example, a workflow implementing a very detailed and structured analysis of performance bottlenecks can be created. Its aim will be to define the best practices for performance analysis by standardizing on a set of performance engineering tools to be used together with the precise sequence in which they should be applied by the users. This will not only provide clear guidelines for novice performance engineers, but will also enhance the work of experienced ones by adding more structure to their daily operations. What is more, having easy to follow suggestions on how exactly analysis techniques should be applied can also improve the adoption of performance engineering tools by a broader HPC community.

Going one step further, PAThWay can also be extended to support the execution of complete, multi-step performance engineering projects. For example, the generic workflow for performance tuning presented in Chapter 15 can be applied to real-world optimization projects as those per- formed by the supercomputing centers. As a result, the quality and the success rate of perfor- mance tuning projects can be enhanced which will inevitably lead to better, more efficient opera- tions for the centers and an increased overall user’s satisfaction.