Tools for Analysis of Performance Dynamics of Parallel Applications

(1)

Tools for Analysis of

Performance Dynamics

of Parallel Applications

Yury Oleynik

Fourth International Workshop on Parallel Software Tools

and Tool Infrastructures

(2)

Outline

• Project Overview

• Performance Dynamics

I. PAThWay

II. Periscope Performance Dynamics Analysis

• Conclusion

(3)

Performance Dynamics of Massively Parallel Codes

• German Research School for Simulation Sciences, Aachen

• Forschungszentrum Jülich

• Gesellschaft für numerische Simulation mbH, Braunschweig

• RWTH Aachen

• Technische Universität Dresden

• Technische Universität München

(4)

Performance Dynamics Aspects

• Runtime performance dynamics

– A

utomatically search for relevant properties of runtime

performance dynamics

• Cross-experiment performance dynamics

– Provide a tool for structuring, automation and tracking of

(5)

PATHWAY

(6)

Problem statement

• Performance engineering is an iterative cycle

– How to organize the whole process? – What tools to use, when and how? – Have the performance improved in comparison to results of the original version from one month ago?

• Optimization spans over months

– What analysis have I done yesterday? – Where have I stored the traces?

• Repetitive, manual processes

– How do I write LoadLeveler script?

Baseline Establish/Update Execute Parallel application Monitor Performance Analyze Bottlenecks Optimize problematic code sections Verify

(7)

PAThWay

• Eclipse plug-in for structured and methodical

performance engineering using workflows

• Goals:

– Manage individual tasks as part of one workflow – Keep track of the performance dynamics

– Automate performance engineering tasks, where possible – Abstract complexity of the underlying software and hardware

(8)

(9)

Workflow specification

• Formal specification of the optimization process using workflows

– Workflows specification using Business Process Model and Notation (BPMN) format

– Predefined components for performance engineering tasks – Predefined workflows:

• Scalability analysis • Cross-platform analysis

– New workflows can be added using graphical editor

• Once defined, workflows can be executed

– Live-tracking of workflow execution

(10)

Workflow components

• Interactive tasks

– Simulation code and perf tools selection – Runtime environment configuration

– Prompting user for documentation input for historical records

• Loops

– Starts all experiments at once

• Domain-Specific Items

– Execution management (creating batch scripts) – Tools configuration and invocation

(11)

Workflow Editor: Scalability Analysis

Workflow editor Available workflow components

(12)

Data and Workflow Repository

• Stores workflow related information

• Contains key properties of the supported tools:

– Characteristics (can profile/trace. etc) – Startup arguments

– Meta data about the tool

• Experiments meta-data:

– Run date/time

– Location of result files – Environment settings – StdOut/Error

(13)

Experiment Browser

Experiments view Database stores also properties of the tools

(14)

Project Documentation

• Yes, you must still invest time in documentation

but …

• PAThWay prepares templates

• Documents each completed experiment

• Automatically inserts relevant experiment meta-data

• User can further extend documentation using

(15)

(16)

Supportive Modules

• Parallel Tools Platform Module

– Starting interactive/batch jobs

– Monitoring execution & accessing data

• Keep snapshots of the sources

– Tracks the code evolution – Based on Git

• Environment Detection

– Captures loaded modules – Captures environment

(17)

PAThWay

• Available as an Eclipse plugin from the update site:

http://periscope.in.tum.de/pathway/eclipse/

• Installation guide:

(18)

AUTOMATIC PERFORMANCE

DYNAMICS ANALYSIS WITH

(19)

(20)

Periscope

• Motivation for Performance Dynamics Analysis

– Location and severity of performance bottlenecks is time-dependent – Performance changes manifest themselves at various time scales – Dimensionality of performance measurements makes manual

investigation by the user tedious

• Analysis goals:

– Automatically detect changes in temporal performance behavior – Quantify the negative impact of performance changes

– Reduce complexity and size of time-dependent measurements – Simplify comprehension (no graphical visualization)

(21)

Periscope

• Helps to answer following typical questions:

– Does the performance degrade over time?

– What is the most relevant performance degradations? – When is the degradation observed?

– Which process/location is impacted by the performance degradation? – Are there similar degradations found in other processes or functions?

• Approach

– Multi-scale analysis

– Qualitative abstraction of time series

(22)

Periscope:

Analysis Steps

1. Measurement collection

2. Preprocessing

3. Qualitative Abstraction

(23)

Periscope:

Measurement

• Periscope uses Score-P to collect dynamic profiles:

Let C be the set of all profiled source code locations within the body of the progress loop, P set of all profiled application process, M set of all

measured metrics and I a sequence of all profiled progress loop iterations, then the we can define time-series Dynamic Profile as a mapping:

which maps a source code location c ϵ C, a process p ϵ P a metric m ϵ M and

(24)

Periscope:

Preprocessing

a) perform scale-space filtering of time series with Gaussian filter

b) detect extremas and inflexion points over scales

c)

connect extremas and inflexion points over scales

a)

(25)

Periscope:

Qualitative Abstraction

a) track extrema and inflexion points from coarse to fine scales

b) label intervals between extrema and inflexion points

c)

extract the level of the tree with the maximum “lifetime”

D AB C C B CD B C AB C C B BCD C BCD CD A D A C B C C B C

a)

b)

c)

A - concave increase B - concave decrease C - convex decrease D - convex increase E - linear increase F - linear decrease G - constant

(26)

Periscope:

Search for dynamics properties

a. Search the maximum “lifetime” level for relevant temporal patterns

Allows flexible specification of the performance dynamics patterns

specified using the following parameters:

– Qualitative shape specification – Perceptual salience

– Magnitude/Duration of the segment – Any combination of parameters

(27)

Example

• Performance properties examples:

– Local maxima (peaks) in Execution time of the main loop body in process 0 are observed during iterations 6-24, 44-47, 57-63, 67-73, 91-104

– The most distinctive local minima (valley) in Execution time of the main loop body in process 0 is observed

during iterations 74-100

– The tallest increase in Execution time of the main loop body in process 0 is observed during iterations 0-21

(28)

Conclusion

• Temporal performance dynamics is important

• There are at least two aspects to consider

– Runtime performance dynamics

– Cross-experiment performance dynamics

• Tools support is needed to help answer questions:

– What are the most relevant runtime performance degradations? – How to organize performance optimization process and track

performance evolution?

• Our solution

– Periscope Performance Dynamics Analysis – automatically search for temporal performance degradations