Monitoring Application Performance User and Reference Guide

(1)

5.6.5 Foglight

®

Monitoring Application Performance

User and Reference Guide

(2)

This guide contains proprietary information protected by copyright. The software described in this guide is furnished under a software license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose other than the purchaser’s personal use without the written permission of Quest Software, Inc.

The information in this document is provided in connection with Quest products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Quest does not make any commitment to update the information contained in this document.

If you have any questions regarding your potential use of this material, contact: Quest Software World Headquarters

LEGAL Dept 5 Polaris Way Aliso Viejo, CA 92656 www.quest.com email: [email protected]

Refer to our Web site for regional and international office information. Trademarks

Quest, Quest Software, the Quest Software logo, Foglight, IntelliProfile, PerformaSure, Spotlight, StealthCollect, TOAD, Tag and Follow, Vintela Single Sign-on for Java, and vFoglight are trademarks and registered trademarks of Quest Software, Inc in the United States of America and other countries. For a complete list of Quest Software’s trademarks, please see http:// www.quest.com/legal/trademark-information.aspx. Other trademarks and registered trademarks are property of their respective owners.

Third Party Contributions

Foglight contains some third party components. For a complete list, see the License Credits page in Foglight online help. User and Reference Guide

December 2012 Version 5.7.1

(3)

Introduction to this Guide

This User and Reference Guide provides an overview of Application Performance Monitoring (APM) with the Foglight Cartridge for Application Operations. It includes conceptual information to help you envision an APM installation with Foglight and an overview of incident management workflows through the Service Operations Console (SOC).

This guide is intended for Application Performance Managers who administer the overall design, implementation, and execution of the application performance monitoring strategy.

About Quest Software, Inc.

Established in 1987, Quest Software (Nasdaq: QSFT) provides simple and innovative IT management solutions that enable more than 100,000 global customers to save time and money across physical and virtual environments. Quest products solve complex IT challenges ranging from database management, data protection, identity and access management, monitoring, user workspace management to Windows management. For more information, visit www.quest.com.

Contacting Quest Software

Refer to our Web site for regional and international office information.

Email [email protected]

Mail Quest Software, Inc.

World Headquarters 5 Polaris Way

Aliso Viejo, CA 92656 USA

(6)

Monitoring Application Performance: User and Reference Guide 6 Introduction to this Guide

Contacting Quest Support

Quest Support is available to customers who have a trial version of a Quest product or who have purchased a Quest product and have a valid maintenance contract. Quest Support provides unlimited 24x7 access to our Support Portal at http://www.quest.com/support.

From our Support Portal, you can do the following:

• Retrieve thousands of solutions from our Knowledge Base • Download the latest releases and service packs

• Create, update, and review Support cases

View the Global Support Guide for a detailed explanation of support programs, online services, contact information, policies, and procedures. The guide is available at: http://www.quest.com/ support.

(7)

1

Introducing Foglight APM

Application Performance Monitoring (APM) refers to a set of capabilities or technologies used by IT staff to support the incident and problem management processes on multi-tier applications. A comprehensive APM implementation generally includes technology that includes the following capabilities:

• End-user experience monitoring

• Run-time application topology and execution path discovery, modeling, and display • Transaction profiling across the technology stack

• Resource consumption and event monitoring within the discovered components • Analytics

Quest’s Foglight APM solution supports all of these features, and provides the information to the right users at the right time through a set of role-oriented operational dashboards.

Although there is no single organizational alignment or uniformly agreed upon set of titles for the actual practitioners of APM, the following role descriptions indicate the types of functions that typically interact with a Foglight APM installation:

Foglight Administrator — responsible for configuring and managing Foglight, and for performing administrative tasks.

Note With the exception of Foglight Administrator, the roles described here are not roles defined in Foglight.

Application Performance Manager — responsible for the design, implementation, and execution of the APM strategy.

Application Support — responsible for ensuring that application services are performing as expected. Identifies and triages service degradations or outages.

Platform Specialist or Administrator — responsible for the operation of an individual application component (for example, a database, a host, an application server, or a web server). Receives and responds to Foglight alarms on their platform or domain.

Application Architect — a senior escalation resource who is typically engaged by Application Support for assistance in locating the source of an application issue.

Application Developer — responsible for the application code. Uses the Foglight

transactional performance dashboards to gain understanding of the operational execution and performance of their code.

(8)

Monitoring Application Performance: User and Reference Guide 8 Chapter 1—Introducing Foglight APM

Understanding the Foglight APM Implementation

Foglight supports a modular approach to APM that allows customers to implement as many or as few pieces of the APM solution as required. This guide is intended as a comprehensive overview of an end-to-end APM installation strategy, and therefore includes cross-domain views.

Before you implement an APM strategy, it is important to perform proper requirement gathering, environment sizing, project planning, and scoping, just as you would with any other application. Quest Professional Services has developed a comprehensive strategy for APM implementation. You can review our process online at:

http://www.quest.com/professional-services/performance-monitoring.aspx.

Foglight Requirements for APM Workflows

In order to make the best use of this document, ensure that your Foglight installation meets the following requirements before beginning the workflows:

• Install the Foglight Management Server and ensure it is running. For more information, see the Foglight Installation and Setup Guide set.

• Install and configure the cartridges required to monitor application components (for example, operating system, application servers, databases). For more information, see the documentation included with the cartridges you are using.

• Configure End-User transaction in Foglight Experience Monitor (FxM), Foglight

Transaction Recorder (FTR), and Foglight Experience Viewer (FxV). For more information, see the documentation for these components.

• Build services, either through Dependency Mapping or manually, and add the services to the Service Operations Console. For more information, see the topic “Creating and Maintaining Services” in the Managing Dependency Mapping User Guide.

Recommended Service Structures

In Foglight, a service is any component or group of components that you want to monitor. If you have the Advanced Operator role, you can create services in the Service Builder dashboard to reflect the components in your monitored environment that are meaningful or interesting to your organization. For more information, see the topic “Monitoring Your Services” in the Foglight User Help.

APM users typically organize their application services into three types: a Top Level Service, Tier Services, and one or more User Services. Each logical tier in the application should have its own service, which includes the application components (for example, databases, application servers, web servers), and the hosts that support the components.

• The Top Level Service contains all elements in the application. It includes all child services and is added to the Service Operations Console (SOC) for Application Support.

• The Tier Service is a logical representation of a service component. Foglight organizes data into default tiers, including User, Web, Application, Database, Host, and Agent. Over time, this Tier Service can track operational level agreements.

(9)

Chapter 1—Introducing Foglight APM

• The User Service contains all End-User artifacts, and organizes the transactions that support the application. For most users, a single End User service is sufficient.

For example, a typical small web application might have the following service structure:

This is a high-level, or tier view of the service structure. This service includes four tiers: user, web, application, and database.

This service can also be represented as:

This is the Application Infrastructure view. Here you can see the hosts contained in each tier: an Apache server (Web tier), two instances of WebLogic each hosted on a separate application server (App tier), and an Oracle database instance (DB tier).

For more information, see “Using the SOC End User Tab” on page 11.

Service Level Agreements in Foglight APM

Foglight automatically examines each service and establishes its availability and service level compliance. By default, a service is available and compliant with its Service Level Agreement (SLA) if it does not have any fatal alarms.

Quest recommends the following general guidelines for APM:

• Implement a top-level SLA based on end-user transactional data to allow the Application Support team to become familiar with SLA functions and the data available. For more information, see “Defining How the SLA is Calculated for Services” on page 10.

• Defer the implementation of tier-based SLAs until your organization is ready to implement official operational level agreements for the infrastructure and component level teams.

(10)

Monitoring Application Performance: User and Reference Guide 10 Chapter 1—Introducing Foglight APM

Defining How the SLA is Calculated for Services

When you create a new, or edit an existing, service, you can specify which components of the service are used to determine its overall availability.

Note This features requires Foglight Management Server 5.6.4 or later.

To define how the SLA is calculated:

1 On the Foglight navigation panel, under Dashboards, click Services > Service Builder. 2 To edit an existing service, click the edit icon for the tier.

Tip To create a new service, click Add A New Category, and review the “Building a Service” topic in the Foglight User Help.

The Create Basic Information page of the Edit Service Wizard opens.

3 In the Objects used to determine the availability of this service section, select one of the following options:

• All components added to this service. This is the default option.

• Only components in selected tiers. Select this option to specify the individual tiers that you want to include. Click the check box beside the tier name to include it.

(11)

2

Using the SOC End User Tab

Service mechanisms in Foglight allow users to organize monitoring components into logical groups. A primary use case for this is for a multi-tier application where objects are organized into service tiers and visualized in the Service Operations Console (SOC).

The Foglight Cartridge for Application Operations includes an extension to the SOC that adds visualization for end-user data in the context of application services in Foglight. This information appears on the End User tab of the SOC. Use this tab to quickly identify if any transactions are having issues, to determine the magnitude of impact on the user population, and to view

preliminary information that can aid in troubleshooting, including geographic discrepancies and whether the performance problems are in the back end architecture or the client-side browser or network.

The following specific end-user monitoring systems can be included:

• FxM — Foglight Experience Monitor. A network appliance that monitors Web transactions from real end users by tapping into the HTTP/HTTPS traffic stream as it transverses the internal network.

• FxV — Foglight Experience Viewer. A transaction repository that stores and indexes every HTTP/HTTPS request/response pair captured from the network. FxV users can search, find, and analyze individual problematic transactions.

• FTR — Foglight Transaction Recorder. A Windows-based application that records and plays back synthetic Web transactions on regular intervals.

Note For complete information about FxM, FxV, and FTR, see their product documentation.

Pre-requisites for Monitoring End User Data in the SOC

In order to make full use of the End User performance monitoring in the SOC, ensure that the following pre-requisites are met:

• The End User Core Cartridge, the Cartridge for FxM, and the Cartridge for FxV have been installed on the Management Server.

Important These cartridges must be installed to enable the End User tab in the SOC, regardless of whether you are collecting data from FxM and FxV.

(12)

Monitoring Application Performance: User and Reference Guide 12 Chapter 2—Using the SOC End User Tab

• The Foglight Experience Monitor (FxM) and/or Foglight Transaction Recorder (FTR) are installed and configured to capture end user data and the appropriate End User cartridges are installed and configured to send end user data to Foglight and visualize it in the browser interface.

• You must create a service that includes FTRResult and/or FxMApplicationResult objects. You can create and edit services using the service builders included with the Cartridge for Application Operations (that is, Build Service from Host Dependencies, Update Service from Dependency Data, or Build Services from Service Suggestions).

For information about building services, see “Creating and Maintaining Services” in the

Managing Dependency Mapping User Guide.

For Foglight cartridge installation instructions, see the Administration and Configuration Help. Important If you choose to manually create a service using the Foglight Service Builder, you must manually

identify and select topology objects. For more information, see “Manually Building a Service” in the Managing Dependency Mapping User Guide.

Configuring Trace Analysis for the SOC

In order to make the trace analysis functionality in the Cartridge for Application Operations useful, you need to create hit filters in FxV.

A hit filter is a collection of match conditions and actions to be performed when a hit matches those conditions. Every hit filter is evaluated against each hit as it enters an FxV Archiver.

Hit filters can be used to detect and alert on any per-hit conditions, to mark interesting hits for later searching, and to manage hit storage. They can also be used to define events within transaction filters.

Hit filters are helpful because they provide a human context to the captured hits. For example, it is easier to search and alert on Login Attempts using a hit filter than to search and alert by the server, path, and fields used by the Web application.

Hit filters are very powerful in their ability to extract custom fields and update metrics. Using regular expressions, hit filters can pull data from fields, headers, cookies, or from page content. In order to ensure that hit filters are coordinated with the FxMApplicationResult, use the FxMApplicationResult name as a filter criterion in FxV. For procedures describing how to configure hit filters in FxV, see “Configuring the Hit Analysis Process” in the Foglight Experience Viewer User Guide.

For more information about trace analysis in the SOC, see “Real User Performance Detail View” on page 86.

Exploring the End User Tab

The End User tab displays a set of tiles representing transactional performance collected from FxM and/or FTR. The type and dynamics of transactional data dictate the number and type of tiles that appear on this tab, along with the Display Options you can configure (see “Configuring Display Options” on page 16).

(13)

Chapter 2—Using the SOC End User Tab

End user transaction groups are divided into:

There are a number of options for controlling how end user transactions are grouped. For more information, see “Configuring Display Options” on page 16.

Note You must add a service containing end user objects (FTRResults and/or

FxMApplicationResults) to the SOC in order for the End User tab to be visible. To access the End User tab:

1 Log in to the Foglight browser interface.

2 On the navigation panel, under Homes, click Service Operations Console. 3 On the Service Operations Console, click the End User tab.

Real Users (FxM and FxV)

(14)

Real Users

When a service includes an FxMApplicationResults model object, it appears on the Real User Tile, providing an at-a-glance view of that transaction’s performance. This tile displays the total number of active end users interacting with the monitored application, transaction front-end and back-end response times, and other metrics. These metrics are collected by FxM.

The Trace Analysis health status icon, located in the bottom right corner, shows that trace analysis metrics are tracking to their historical norms. Deviations from those norms indicate an anomaly that should be investigated.

For more information about the metrics appearing on this tile, see “Real User Tile” on page 77. Drill down on this tile (click the title bar) to open the Real User Performance Detail View. For more information about the metrics appearing on this view, see page 86.

The tabs available in this view depend on the monitoring capabilities of your Foglight installation. For example:

(15)

• The Response Time Breakdown Tab requires the Cartridge for Java EE Technologies version 5.8.2 or later. For more information, see “Requirements for Using the Response Time Explorer Dashboard” on page 28.

• The SLA Violators Tab requires FxM and a deployed and configured FxM Map Agent. For more information, see “Requirements for Using the Geographical Perspective Dashboard” on page 18.

Synthetics

When a service includes an FTRResults object, the End User tab displays one or more Synthetics tiles. These metrics are collected by Foglight Transaction Recorder (FTR). Depending on the number of tiles and the configured Display Options (see “Configuring Display Options” on page 16), a collection of Single-location Synthetic Layout or Multi-location Synthetics Layout tiles appear.

Single-location Synthetic Layout

The Single-Location Synthetics Tile shows summarized data about a synthetic transaction executed from a single location. This type of layout shows the number of script executions and their execution times over the monitored period, along with other metrics. For more information about the metrics appearing on this tile, see page 81.

.

Drill down on this tile (click the title bar) to open the Synthetic Result Detail View. For more information about the metrics appearing on this view, see page 90.

Multi-location Synthetics Layout

The Multi-Location Synthetics Tile shows summarized data about a synthetic transaction executed from multiple locations. This type of layout shows the total number of script execution locations and the numbers of locations for each alarm state (Normal, Warning, Critical, and Fatal). For more information about the metrics appearing on this tile, see page 80.

(16)

Drill down on this tile (click the title bar) to open the Synthetic User Performance Detail View. For more information about the metrics appearing on this view, see page 91.

Drilling down on an individual location displays the Synthetic Result Detail View. For more information about the metrics appearing on this view, see page 90.

Configuring Display Options

You can customize the layout of the End User tab by selecting the appropriate options in the Tile Display Options dialog box. Use these options to control the total number of tiles, the order in which they appear, and the roll up options for multiple FTR locations.

Note These settings apply on a per-user basis.

To configure display options:

1 On the Service Operations Console, on the End User tab, click Display Options. The Tile Display Options dialog box opens.

(17)

2 Configure these options, as required.

• The Enable Grouping option groups user tiles together by transaction name. By default, groups are sorted by their severity first, then by their name. This grouping is not affected by the Sort Order options.

• The Show at most n tiles or groups setting determines the total number of tiles displayed. Synthetic transaction locations are first rolled up using the rollup criteria below. After that, the total number of tiles is truncated based on the maximum number of tiles that you specify. • You can roll up synthetic transactions locations into multi-location FTR (Synthetics) tiles

based on these criteria, in the following order:

• the maximum number of single-location Synthetics tiles, using the Rollup if there are more than n FTR tiles setting, and then

• the maximum number of locations per script, using the When there are more than n location(s) for a script setting.

• Use the Sort Order options to sort the tiles by either type (Real End Users first) or alarm severity (Sort by Alarm State).

For example, you can configure the End User tab to display a maximum of ten tiles and sort them by the alarm state (highest severity first).

3 Click Save.

The Tile Display Options dialog box closes and the End User tab refreshes, reflecting the newly updated display options.

(18)

3

Understanding Geographical Perspectives

The Geographical Perspective dashboard displays the real end user activity for the top 100 locations for a selected application component based on performance problems and errors, total traffic, response time, and other factors. It provides a list of performance-related measurements (such as, “Show locations with the most: Hits” or “Show locations with the highest: Hit Response Times”) that help you quickly and easily find patterns and problems that have geographical features.

Requirements for Using the Geographical Perspective Dashboard • Foglight Experience Monitor (FxM) Appliance

• FxM Agent configured and actively collecting data • Foglight Management Server, version 5.6.5 or later

• Cartridge for FxM and End-User Core Cartridge installed on the Management Server • Cartridge for Application Operations, version 5.7 or later

(19)

Chapter 3—Understanding Geographical Perspectives

Creating a Map Agent

To visualize real end user activity on the map, you must first configure a map agent. A map agent monitors a single FxM Appliance. It must be paired with an existing Foglight Real User

Monitoring for Web agent to import geographical information into Foglight. You can only deploy one map agent per Foglight Real User Monitoring for Web agent instance.

To create a map agent:

1 On the navigation panel, under Dashboards, click End User > Geographical Perspective. 2 The Geographical Perspective for ApplicationName dashboard opens in the display area. 3 Select an FxM Application from the FxM Application Result Table located at the bottom of

the navigation panel.

4 On the Geographical Perspective dashboard, click Create a map agent to import geographic data. The Map Agent Setup wizard opens.

The map agent requires an existing FxM agent that is active and collecting data from a Foglight Experience Monitor (FxM) Appliance.

(20)

Foglight: Monitoring Application Performance Guide 20 Chapter 3—Understanding Geographical Perspectives

Important If your FxM Appliance is at or near capacity, or if it is experiencing packet drops, it is recommended that you do not deploy the map agent. The map agent causes a small but measurable impact on the Appliance.

5 Verify that your FxM Appliance and agent are in a healthy state and click Next. 6 On the Advanced Agent Settings page, specify the agent settings.

a Select the granularity levels at which you want the map agent to import data. You can select any combination of: Hourly, Daily, Weekly, or Monthly. By default, only Hourly and Daily are selected.

(21)

b Specify the location where the map agent is deployed. By default, this is the same Agent Manager that is hosting the FxM agent. Select an alternate location from the drop-down list if necessary.

7 Click Finish.

8 On the Geographical Perspectives for ApplicationName dashboard, click Import geographical data to import the data for that application from FxM to Foglight. A message box opens.

9 Review the health of your Appliance.

Note If the load on your FxM Appliance is too high due to other geographic data collections that are currently running, you can deactivate those collections to reduce the load. Click the text link: Click here to deactivate or reactivate geographical data import for other

applications.

10 Click OK. The map agent imports geographical data for the selected application. The dashboard refreshes, displaying the relevant end user activity.

Note This process may take some time, depending on the amount of data being imported. If there is no data available for your selected time period, the aggregation interval and timeslice fields will appear empty.

If necessary, you can edit the map agent properties through the Agent Status dashboard. For more information, see “Editing Map Agent Properties” on page 25.

(22)

Viewing Real End User Activity

The Geographical Perspective dashboard is divided into three panes. The left-hand pane contains lists of: available aggregation intervals, timeslices, and measurements. The center pane displays either a map, if the Map tab is selected, or a list of results in a tabular format, if the List tab is selected. The right-hand pane provides graphical results from the measurements, and the option to drill down for more information.

Note If you are using a version 5.6.4 Management Server, only the List tab is available. The Map tab requires Management Server 5.6.5 or later.

If you are using Internet Explorer (IE) 8 or earlier, only the List tab is available. To view real end user activity in a geographical perspective:

1 On the navigation panel, under Dashboards, click End User > Geographical Perspective. 2 Select an FxM application component from the FxM Application Result Table at the

bottom of the navigation panel.

3 Select a Measurement from the list to view the related transaction data. The map and right pane update based on your selection.

For example: select Show locations with the highest: Page Response Times.

The map updates to indicate the locations that are experiencing the highest average page response times in the selected timeslice. The right pane displays a chart of the page

processing time for the entire application (all locations), and the top five locations are listed in a table below the graph and timeslice selector.

Double-click anywhere on the map to zoom in on that location. You can also use the zoom tool in the map pane to zoom in or out. You can move the map in any direction by clicking and holding the left mouse button and dragging the cursor.

(23)

The fill-level of each location indicator on the map shows how close that location is to the maximum value in the current timeslice. In this example, Georgia has the highest response time, so its location indicator appears full.

4 Select a timeslice from the Timeslice list, or use the forward or back arrow buttons to step through the available timeslices.

Tip The vertical blue bar on the graph indicates the currently selected timeslice.

5 Hover your cursor over a location indicator to open a quick view of metrics for that location.

6 Click an individual location indicator to view a graph of the average page response time for that location in the right pane.

The Similar Locations list in the right pane updates to show locations that are ranked near the selected location in the current category.

(24)

that are also exhibiting the performance issues that the selected location is experiencing. Click any location name in the list to view the graph for that location.

7 Click Location Detail to drill down to detailed information for the selected location.

The Location Detail view provides a table of measurements in context that helps you understand how the selected location is behaving relative to other locations during the selected timeslice and aggregation period.

Measurements for the selected location are sorted in order of severity.

The value in the Highest Rank column indicates the highest ranking that the selected location attained at any point during the selected time range. The time range may span multiple timeslices. For example, at some time during the selected time range, New York had the highest page response time of all locations. Therefore, for the measurement Page Response Time, New York’s Highest Rank is 1 (first).

The value in the Current Rank column indicates the selected location’s current ranking for the selected timeslice. This may or may not be the same as its highest rank, depending on when the peak value for the location occurred. In this example, New York is ranked second for the Current Rank for Page Response Time. That is, during the currently selected timeslice, New York has the second highest page response time.

8 Click any location’s name in either the Similar Locations list or the Top Ranked Location column to open a detail view for that location for comparison.

9 To view the transaction data for the selected location in the FxM Appliance, click More Details.

(25)

Editing Map Agent Properties

After you have created a map agent, as described in “Creating a Map Agent” on page 19, you can edit its properties from the Agent Status dashboard. For example, you may want to change the available aggregation granularity, or change the scheduled collection interval.

To edit map agent properties:

1 On the navigation panel, under Dashboards, click Administration > Agents > Agent Status.

2 Select the map agent in the list and click Edit Properties.

Tip Map agents have the type MapFxMAgent and, by default, their names begin with Geo@. 3 Click Modify the private properties for this agent.

The settings become editable.

4 Ensure that the settings for the FxM Appliance database IP address and password match those that the FxM agent is using to access the database.

5 The Associated FxM Agent Name must match the name of the FxM agent exactly. If this box is blank, or if the value does not match that of the FxM agent, the map agent cannot access the FxM data to populate the geographical perspectives.

6 The Locations Per Measurement setting determines the number of locations from which real end user activity is collected for display on the Geographical Perspective dashboard. By default, this value is 100.

(26)

7 The Maximum SQL Retries setting controls the number of times the map agent tries to reconnect to an appliance that is unreachable before stopping data collection. The default is twelve attempts.

8 The five True/False settings allow you to specify the granularity of the data aggregation. Click True for each of the FxM reporting intervals you want to make available: five minutes, hourly, daily, weekly, or monthly. By default, the five minute, weekly, and monthly intervals are disabled (set to False).

Important If you have a large number of application components, using the five minute interval can cause a heavy load on the FxM Appliance, negatively impacting performance.

9 The FxM Application Collection list controls which of the available application components the map agent collects data from.

• To edit the default list, click Edit. Select or clear the check boxes for the applications and click Save Changes.

or

• To create a new list without affecting the default list, click Clone. Type a name for the cloned list and click OK.

To edit the cloned list, click Edit. Select or clear the check boxes for the applications and click Save Changes.

10 The Data Collection Scheduler setting controls how frequently the map agent checks the FxM Appliance for data. By default, the collection interval is fifteen (15) minutes.

Note The agent imports all data for the selected granularities up to one hour old regardless of the collection interval.

To edit the default schedule, click Edit.

To create a new schedule without affecting the default schedule, click Clone. 11 When you are finished setting the agent properties, click Save.

Generating Geographical Perspective Reports

There are a number of reports specific to geographic perspectives. For example, you can generate reports on top locations by errors and violations, by response time, or by user volume.

To access these reports:

(27)

The report wizard prompts you to specify the input parameters (such as time range, application, and granularity) and properties (such as report name and export format). You can also schedule the report to run at regular intervals.

For more information about a specific report, run the report wizard from the Geographical Perspectives dashboard.

(28)

4

Examining Response Times

The Response Time Explorer dashboard displays the sampled requests with the longest execution times for a selected end user session during a selected timeslice. The requests are listed by execution time. Use this dashboard to quickly see where request bottlenecks are occurring, and where in the application infrastructure the time is being spent. With the Timeslice navigator, you can view the historical average response times in five minute intervals.

Requirements for Using the Response Time Explorer Dashboard

• Foglight Management Server, version 5.6.4.2 or later, with the following cartridges installed and configured:

• Cartridge for Application Operations, version 5.7.1 or later • Cartridge for Java EE Technologies, version 5.8.2 or later

(29)

Chapter 4—Examining Response Times

• Cartridge for FxM, version 5.6.2 or later • Cartridge for FxV, version 5.5.8.1 or later • End-User Core Cartridge, version 5.6.3 or later • Foglight Experience Monitor (FxM) Appliance

• Foglight Experience Viewer (FxV) Appliance, version 5.6.5 or later

Configuring Request Traces

The Cartridge for Java EE Technologies provides the ability to trace requests. You must configure request tracing through the Java EE Agent recording.config property file. For detailed information about configuring request tracing, see the Managing Java EE Systems: Installation and

Configuration Guide.

In particular, you must enable the IndividualRequestBreakdowns option in order for the request details to be matched to the appropriate end user results.

To enable IndividualRequestBreakdowns:

1 On the navigation panel, click Administration > Agents > Agent Properties. 2 In the Namespace > Type column, expand the JavaEE module (click the + sign). 3 Select recording.config.

4 Locate the section that begins with: # Rules for the collection of details on sampled requests. 5 Configure the request detail rules to include IndividualRequestBreakdowns.

For example: RequestDetailRules = { /.*medrec.*/ : ("IndividualRequestBreakdowns"), }; RequestDetailDefault = ("IndividualRequestBreakdowns"); 6 Click Save.

Associating FxV Hit Filter Mappings

In order to present a comprehensive view of requests as they relate to end user results, you must create a link from the FxV hit filter to the Java request and to the FxM (end user) application result.

For detailed information about configuring FxV hit filters and FxM application results, see the documentation included with the Foglight Experience Viewer and Foglight Experience Monitor Appliances respectively.

The response time breakdown only applies to the specific requests that pass through an

instrumented application server. The Front End/Back End time breakdown shown on the Real User Tile applies to complex multi-hit page objects that match the FxM filter. If there is a substantial difference in displayed response time between the Real User Tile and the response time

(30)

Foglight: Monitoring Application Performance Guide 30 Chapter 4—Examining Response Times

By default, if no mapping is provided and an existing hit filter has the exact same name as the transaction, that filter is used. The application result is mapped automatically.

To associate a hit filter to an application result:

1 On the Service Operations Console (SOC), click the title bar of the Real User Tile of the application result you want to work with.

2 The Real User Performance Detail View opens, with the Response Time Breakdown tab displayed.

3 Click Associate Hit Filter Mappings. The Associate FxV Hit Filter Mappings dialog box opens.

4 Click Add. The Associate FxV Hit Filter Wizard opens.

5 Select the hit filter that matches the application result.

6 Optional — click Next to configure proxy objects. For more information about assigning proxy objects, see “Assigning Proxy Objects” on page 35.

(31)

7 Click Finish.

The associated hit filter appears in the list in the Associate Hit Filter Mappings dialog box.

Exploring Response Time Breakdowns

The Real User Performance Detail View provides an at-a-glance overview of response time breakdowns for a selected real user transaction group.

To view the Response Time Breakdowns:

1 On the Service Operations Console (SOC), click the title bar of any real user transaction group that has an associated hit filter.

Tip Real user transaction groups that are configured for response time breakdown views are indicated with the icon at right:

The Real User Performance Detail View opens. 2 Click the Response Time Breakdown tab.

Here you can quickly check the execution time for a request through the various components that are included in the transaction group.

Note The execution times in this view are aggregated for the current timeslice.

The upper portion of the tab provides a list of components with average execution times for the current timeslice and a sparkline that visualizes the average execution time through that component for the selected time range.

(32)

The lower portion of the tab provides the same average execution time information in a

dependency path view, with the average execution time through each component displayed below the icon for the component.

The dark grey arrows indicate the path taken by at least one request in the current timeslice (for example, between 3:20 and 3:25pm).

The light grey arrows indicate the path taken by at least one request in the selected time range (for example, in the last four hours).

For example, in the previous image, you can see that 181.9 ms were spent on MedRec1Server1, and only 79.9 ms were spent on MedRec1Server2. The breakdown information makes it clear that

MedRec1Server1 is causing a bottleneck for requests.

Click a component icon to view a chart of the average historical response time for that component based on the selected time range.

This chart provides a comparison of the average execution time for the selected component (in blue), overlaid on the total average execution time for the transaction (in grey). The vertical blue bar on the chart indicates the current timeslice.

Tip If you click a datasource component icon, the details view also includes a Map to Application Component button. For more information, see “Mapping Datasources” on page 35.

Click More Details to drill down to a detailed view of the selected application component. For example, from an application server component, you can access the Java server details view.

(33)

Viewing the Response Time Breakdown Explorer

Click Explore Breakdowns on the Real User Performance detail view to drill down for more information.

The Response Time Breakdown Explorer dashboard opens.

This dashboard displays the top sampled requests (that is, those with the longest execution time) for a selected transaction (a mapped FxMApplicationResult) during a selected timeslice. The requests are listed by execution time, along with an indication of where in the path the bottleneck occurred, and the time the request was executed.

Execution Time History

The charts on the left-hand side provide a comparison of the average execution time for the component (in blue), overlaid on the total average execution time for the transaction (in grey). The vertical blue bar on the chart indicates the current timeslice.

Use the Timeslice navigator to step through the selected time range (for example, four hours) in five minute intervals.

To view the execution times for a different transaction:

• Open the navigation pane and select a transaction from the Breakdown for Applications list.

(34)

Top Sampled Requests by Execution Time

The right-hand side of the dashboard provides a list of the sampled requests with the top (longest) execution times, and a dependency path view that allows you to trace the request through the components. Select a request from the list to update the dependency path view.

Blue arrows mark the path of the currently selected request. For example, in the image above, the selected request (medrec/admin/viewNewlyReigsteredPatients.action) begins at the end user, passes through the medrecweb proxy object, and reaches the MedRec1Server2 component. It does not reach the database.

The dark grey arrows indicate the path taken by at least one request in the current timeslice (for example, between 3:20 and 3:25pm).

The light grey arrows indicate the path taken by at least one request in the selected time range (for example, in the last four hours).

Click a component’s host name to drill down for more information about that host. From the Response Time Breakdown Explorer you can:

• View Transaction Replay — Opens the hit and session that match the selected request in the FxV Appliance. Here you can replay the content of the hit as the affected end user saw it to determine exactly where an issue occurred.

For more information about the Hit Inspector, see the Foglight Experience Viewer User Guide.

• Request Details — Opens the Java EE Requests view, which allows you to review the details of the request, including calls completed and incomplete, response time current, average, and maximum, and exceptional exits.

You can also collect traces from this view (click the Action icon for the request and select Collect Traces). For more information, see “Monitoring Requests” in the Monitoring Custom Applications User and Reference Guide.

• Examine Java Trace — If you have collected traces for a request, you can drill down to the trace details.

(35)

For more information about the Traces View, see “Managing Traces” in the Monitoring Custom Applications User and Reference Guide.

Additional Configuration Options

There are additional configuration options for exploring response time breakdowns that may be useful in some instances. For example, if you are associating hit filter mappings to a transaction that contains a proxy object, review the topic Assigning Proxy Objects. If your dependency path view contains a datasource that needs to be mapped to a specific component, review the topic Mapping Datasources.

Assigning Proxy Objects

In some cases, requests may pass through one (or more) proxy objects, such as a firewall, load balancer, or router, before they reach the application server tier. If you know that this is the case in a transaction group for which you want to view response times and map a hit filter, you can create a proxy path.

To select a proxy path:

1 In the Real User Performance Detail View, click Associate Hit Filter Mappings.

Note If the transaction is mapped to the default hit filter, you must first remove the default hit filter, then add it manually in order to assign proxy objects.

2 In the Associate Hit Filter Mappings dialog box, click Add. 3 Select the FxV Hit Filter from the list. Click Next.

4 On the Proxy Path Selection page, select the option that accurately reflects the transaction path you are trying to map: No hosts, One host, or Multiple hosts between the transaction and application tier.

5 Click Next.

6 Select the proxy object from the list.

Tip If you know the name of the object, you can use the search box to quickly locate it. 7 Click Next.

8 On the Summary page, click Finish.

9 The list in the Associate Hit Filter Mappings dialog box updates. Close the dialog box.

Mapping Datasources

By default, datasources appear on the Response Time Breakdown tab with limited information. If you know the proper type of a datasource (for example, if it is an Oracle database), you can map the datasource to a specific application component. A correct mapping provides access to drill down information, such as host or database metrics.

(36)

To map a datasource:

1 In the Real User Performance Detail View, click the icon for the unmapped datasource.

2 In the details view that opens, click Map to Application Component. The Assign Resolver Wizard opens.

Select whether to map the object to a host, to an application component, or to unassign the current mapping so that you can select a new host or component mapping.

To assign a host object:

1 Select Hosts and click Next.

2 Select a host from the list and click Next. 3 Click Finish.

Now when you click the object’s icon, the Host View opens.

To assign an application component object:

1 Select Application Components and click Next. 2 Select an application component from the list.

(37)

Tip If you are uncertain of the correct application component topology object name, see “Topology Object Name Reference Table” on page 93.

3 Click Next. 4 Click Finish.

Now when you click the object’s icon, the database view opens.

To unassign the current mapping:

1 Select Unassign Current Mapping. 2 Click Next.

3 Click Finish.

The mapping for the object is removed. To assign a different mapping, click Map to Application Component and select a new mapping.

(38)

5

Using the SOC for APM Triage

In order to support triage use cases, Foglight APM provides unique views that cover the intersection of transactions, application components, and infrastructure elements that define an application. The Service Operations Console is the primary dashboard for performing incident management workflows that terminate with a handoff to application, domain, or platform specialists.

A user with the application support role for their organization uses the SOC to visually assess the current state of one or more applications. If transactional issues are present, they examine the state of transactions supported by a particular application, and assess the impact on the user population. They then visualize and evaluate the state of supporting applications and infrastructure to isolate the probable source of the issue. Finally, they engage the appropriate specialist.

This chapter presents use cases that illustrate the types of issues you may encounter. Follow the walkthroughs for these use cases to gain a greater understanding of how Foglight simplifies the triage process. These walkthroughs demonstrate problems arising from the following sources:

• Database Layer • Application Layer • Virtual Machine

Database Layer

Begin all investigations by looking at the multi-application table at the top of the SOC dashboard. Here you can see an overview of the service or tier, including name, service level compliance, and user health status.

Only one service is shown in this example, but in a typical environment, there would be several. Tip You can add and remove columns to customize the information that appears in this table by

(39)

Chapter 5—Using the SOC for APM Triage

To triage an application:

1 Review the Service Level Compliance and User columns of the table to locate the tier of the service that is experiencing performance degradation.

In this case, the User tier icon indicates an issue for users.

The service level compliance icon indicates that the service has not yet violated the service level compliance policy in Foglight. However, if the issue persists, this icon may change to indicate a violation.

2 Review the End User tab to determine the impact on users.

End user tiles are grouped by transaction: MD1Patient (top row), MD1Physician (middle row), and MD1Admin (bottom row). They are also ordered by severity (that is, the users with the worst health status or most alarms appear at the top).

Here you see that one transaction is having an issue, with both the real user tile and synthetic tiles reflecting an issue when accessing MD1Patient. The users accessing MD1Physician and

MD1Admin are not experiencing any issues (their status is green across the board), which indicates that the issue is not affecting the whole application, only the MD1Patient

transaction.

The Trace Analysis health status ( ) icon on the MD1Patient Real User tile indicates that there is a problem somewhere in the trace analysis metrics.

Foglight has also generated critical level alarms ( ) for all four MD1Patient transactions. Since all synthetic transactions are affected, this is not a location-specific issue. Synthetic transactions can be used as a benchmark for healthy performance. In this case, the fact that the synthetics are affected indicates that this is not a problem caused by a one-time error from a single user; it is a recurring or ongoing problem affecting all users.

(40)

Monitoring Application Performance: User and Reference Guide 40 Chapter 5—Using the SOC for APM Triage

3 Click the title bar of the affected Real Users tile to drill down for more information. The Real User Performance detail view opens.

This detail view captures several key metrics in chart format. Consider the following: Metric Key Insight

Response Time Beginning at 09:35, users are experiencing a longer response time. Page Requests At 09:35, the number of page requests is slightly lower than earlier

peak loads. This indicates that the problem is not due to a spike in user activity.

SLA/OLA

Attainment Also at _{Level Agreement (OLA) are still being met, but the percentage is}09:35, the Service Level Agreement (SLA) and Operating decreasing.

Front End /

Back End The time the transaction spent in the back end (that is, in the _{supporting architecture) has increased, and the problem appears to} have started shortly before users were affected (that is, before 09:35).

(41)

Note Click the title bar of the MD1Patient Hits Over 7 Seconds chart to drill down into the individual sessions recorded by FxV. The FxV details allow you to see the Hit URLs and error messages, and to drill down into individual sessions, where you can step through a session to locate the error point.

In this case, drilling down to the sessions shows that this is a content error issue. Some users are seeing error messages in their web browsers. This increases the priority of this issue.

This looks like a serious problem that is impacting many users, and the issue seems to be originating in the back end.

Close this view by clicking the ‘x’ in the upper right corner.

4 Next, investigate the application topology to determine where in the supporting architecture the problem originates.

Click the Dependencies tab to view the application topology. Tip Always work from left to right when triaging dependencies.

Overall, the infrastructure is in good health — all of the hosts have a green check mark, indicating that their status is normal. However, several of the tiers that contain platform and code components appear to have issues that must be investigated.

Trace Analysis The overall number of hits has decreased, as shown in the MD1Patient Hits chart.

The number of users experiencing delay of more than seven seconds in their transaction execution has increased, as shown in the

MD1Patient Hits Over 7 Seconds chart. Metric Key Insight

(42)

The Web tier has no issues. Moving to the right, you see that there is a warning on the Application (MedRecApp) tier.

Tip Hover the mouse pointer over a host icon to open a popup view of the application components.

5 Click the application tier title bar to drill down for more details.

The MedRecApp Details view opens. The banner section of the Summary tab displays the service level compliance for the tier. The lower portion of the view contains a tile for each application component, grouped by host.

Tip For more information about the metrics displayed on views and tiles, see “Reference” on page 67.

(43)

On the Request Types tab, you see that POST/medrec/patient/viewPatient.action is the source of the warning. Select this request.

Tip Use the Search box to filter the list of requests. Type medrec/patient to include only those requests with that string in the name.

The response time for this request has increased and remains high. The Bottleneck Tier Name column lists the database tier as the source of the bottleneck.

Note There are additional columns that may be of interest that are hidden by default. To show them, click the customizer icon to the right of the Search box and select the columns you want to display.

The Execution Time chart at the bottom of the Application Details view (below the list of request types) indicates that the most time is spent in the database tier. It also shows that the time spent in the database tier began to increase at 09:30, which corresponds to the increase in the time spent in the back end that appeared in the Real User Performance view in step 3. This evidence all points to a problem in the database tier, which is where you should look next.

Click the ‘x’ in the upper-right corner to close the Application Details view. Close the

MedRecApp Details view as well.

7 On the Dependencies tab, click the title bar of the MedRecDB database tier to drill down for more details. The details view for the database tier opens.

(44)

There is a critical alarm on the application component tile for the Oracle database. The Oracle Tile displays the top four session bottlenecks as color-coded bars. The larger the main color bar, the higher the percentage of resources were spent on that metric. The thin blue lines on the top of each bar indicate the range for the normal state. The thin green/ yellow/orange bar below the main color bar indicate the metric’s trend compared against itself.

In this case, the database has Lock wait issues.

Now that you have determined the most probable source, you can engage the database administrator to evaluate the problem.

In this scenario, the database and application administrators should have been aware of the problems before the Application Performance Manager became involved. If the administrators were using Foglight, they would have received alarms generated and sent by Foglight.

(45)

Application Layer

Begin all investigations by looking at the multi-application table at the top of the SOC dashboard. Here you can see an overview of the service or tier, including name, service level compliance, and user health status.

Tip You can add and remove columns to customize the information that appears in this table by clicking the customizer icon to the right of the Search box.

1 Review the Service Level Compliance and User columns of the table to locate the service that is experiencing performance degradation.

In this case, the User tier icon indicates an issue for users. The Service Level

Compliance icon is green, indicating that the service level compliance policy in Foglight is being met.

(46)

Tip You can change the display options to roll-up single synthetic transaction locations into multi-location tiles, as shown above. For more information, see “Configuring Display Options” on page 16.

Here you see that Real Users are experiencing an issue when accessing MD1Physician, but the problem is only visible in the Trace Analysis. Synthetic transactions are not experiencing this problem.

The Trace Analysis health status ( ) icon on the MD1Physician Real User tile indicates that there is a problem originating in a traced request.

The OLA (Operating Level Agreement) and SLA (Service Level Agreement) values have degraded slightly. This suggests there are some outliers that are violating the compliance policy in Foglight.

3 Click the title bar of the affected Real User tile to drill down for more information. The Real User Performance view opens.

(47)

This detail view captures several key metrics in chart format. Consider the following: Metric Key Insight

Response Time Starting at 06:30, the response time has increased slightly, but the increase is constant.

Page Requests The number of page requests increased slightly before the response time increased, and has remained high. This indicates an increased load, not just a single spike in user activity.

SLA/OLA

Attainment The Service Level Agreement (SLA) and Operating Level Agreement _{(OLA) are still being met. The majority of users are not experiencing} any problems.

Front End /

Back End The time spent in both the front end and back end (infrastructure) has _{increased, with the larger portion of the time being spent in the back} end.

(48)

Tip Click the MD1Physician Hits Over 7 Seconds chart to drill down to the user sessions in FxV.

The evidence from the metrics indicates that a small number of users are consistently experiencing problems.

Close this view by clicking the ‘x’ in the upper right corner.

Next, investigate the application topology to determine where in the supporting architecture the problem originates.

4 Click the Dependencies tab to view the application topology. Tip Always work from left to right when triaging dependencies.

Overall, the infrastructure is in good health — all of the hosts have a green check mark, indicating that their status is normal. However, the tiers that contain platform and code components appear to have issues that must be investigated.

5 Click the MedRecApp application tier title bar to drill down for more details.

Tip If one of the hosts was experiencing an issue, you could click the host icon to drill down into a Host Detail View instead of looking at the tier view.

The MedRecApp Details view opens. The banner section of the Summary tab displays the service level compliance for the tier. The lower portion of the view contains a tile for each application component, grouped by host.

Trace Analysis The MD1Physician Hits Over 7 Seconds chart shows the number of users experiencing a delay of more than seven seconds in their transaction execution has increased and remains high.

This indicates that a handful of outlier transactions are severely impacting these users, but other users remain unaffected. Metric Key Insight

(49)

Here you can see that both nodes in the WebLogic Server cluster (MedRec1Server1 and

MedRec1Server2) are experiencing a problem with requests. The hosts are unaffected. 6 Click the title bar of the MedRec1Server2 application component tile to open the Application

(50)

On the Request Types tab, you see that POST/physician-web/physician/ createRecord.action is the source of the warning. Select this request.

Tip Use the Search box to filter the list of requests. Type physician to include only those requests with that string in the name.

The response time for this request has increased and remains high. The Bottleneck Tier Name column lists the MedRec1Cluster tier as the source of the bottleneck.

The Execution Time chart below the list of request types also indicates that the most time is spent in the MedRec1Cluster of the application tier.

This evidence points to a problem in the Java code, but requires further investigation. 7 Now that you have determined the probable source of the issue that is affecting users, you

can engage the domain expert to investigate further. The domain expert can use the Custom Applications (Java) dashboards to examine the request and traces in greater detail.

(51)

Virtual Machine

Begin all investigations by looking at the multi-application table at the top of the SOC dashboard. Here you can see an overview of the service or tier, including name, service level compliance, and health status for selected tiers.

Tip You can add and remove columns to customize the information that appears in this table by clicking the customizer icon to the right of the Search box.

1 Review the Service Level Compliance, User, and App columns of the table.

In this case, the User tier icon indicates an issue for users. The Service Level

Compliance icon is green, indicating that the service level compliance policy in Foglight is still being met.

The App tier icon indicates there are issues somewhere in the application tier as well. 2 Review the End User tab to determine the impact on users.

Monitoring Application Performance User and Reference Guide

5.6.5

Foglight

®