Java SE Performance Tuning Revision A

(1)

Java™ SE Performance Tuning Revision A

(2)

This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any.

Third-party software, including font technology, is copyrighted and licensed from Sun suppliers.

Sun, Sun Microsystems, the Sun logo, the Duke logo, Java, JavaServer Pages, JSP, Java HotSpot, Java VisualVM, Sun Fire T1000, Sun Fire T2000, UltraSparc, NetBeans, NetBeans Profiler, Sun Studio, and JavaScript are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries.

The OPEN LOOK and Sun Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds a non-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Sun's licensees who implement OPEN LOOK GUIs and otherwise comply with Sun's written license agreements.

Federal Acquisitions: Commercial Software Government Users Subject to Standard License Terms and Conditions

Export Laws. Products, Services, and technical data delivered by Sun may be subject to U.S. export controls or the trade laws of other countries. You will comply with all such laws and obtain all licenses to export, re-export, or import as may be required after delivery to You. You will not export or re-export to entities on the most current U.S. export exclusions lists or to any country subject to U.S. embargo or terrorist controls as specified in the U.S. export laws. You will not use or provide Products, Services, or technical data for nuclear, missile, or chemical biological weaponry end uses.

DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS, AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

(3)

Ce produit ou document est proté gé par un copyright et distribué avec des licences qui en restreignent l'utilisation, la copie, la distribution, et la dé compilation. Aucune partie de ce produit ou document ne peut ê tre reproduite sous aucune forme, par quelque moyen que ce soit, sans l'autorisation pré alable et é crite de Sun et de ses bailleurs de licence, s'il y en a.

Le logiciel dé tenu par des tiers, et qui comprend la technologie relative aux polices de caractè res, est proté gé par un copyright et licencié par des fournisseurs de Sun.

Sun, Sun Microsystems, the Sun logo, the Duke logo, Java, JavaServer Pages, JSP, Java HotSpot, Java VisualVM, Sun Fire T1000, Sun Fire T2000, UltraSparc, NetBeans, NetBeans Profiler, Sun Studio et JavaScript sont des marques de fabrique ou des marques dé posé es de Sun Microsystems, Inc. aux Etats-Unis et dans d'autres pays.

L'interfaces d'utilisation graphique OPEN LOOK et Sun a é té dé veloppé e par Sun Microsystems, Inc. pour ses utilisateurs et licencié s. Sun reconnaît les efforts de pionniers de Xerox pour larecherche et le dé veloppement du concept des interfaces d'utilisation visuelle ou graphique pour l'industrie de l'informatique. Sun dé tient une licence non exclusive de Xerox sur l'interface d'utilisation graphique Xerox, cette licence couvrant é galement les licencié s de Sun qui mettent en place l'interface d'utilisation graphique OPEN LOOK et qui en outre se conforment aux licences é crites de Sun.

Lé gislation en matiè re dexportations. Les Produits, Services et donné es techniques livré s par Sun peuvent ê tre soumis aux contrô les amé ricains sur les exportations, ou à la lé gislation commerciale dautres pays. Nous nous conformerons à lensemble de ces textes et nous obtiendrons toutes licences dexportation, de ré -exportation ou dimportation susceptibles dê tre requises aprè s livraison à Vous. Vous nexporterez, ni ne ré -exporterez en aucun cas à des entité s figurant sur les listes amé ricaines dinterdiction dexportation les plus courantes, ni vers un quelconque pays soumis à embargo par les Etats-Unis, ou à des contrô les anti-terroristes, comme pré vu par la lé gislation amé ricaine en matiè re dexportations. Vous nutiliserez, ni ne fournirez les Produits, Services ou donné es techniques pour aucune utilisation finale lié e aux armes nuclé aires, chimiques ou biologiques ou aux missiles.

LA DOCUMENTATION EST FOURNIE "EN L'ETAT" ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L'APTITUDE A UNE UTILISATION PARTICULIERE OU A L'ABSENCE DE CONTREFAÇ ON.

(4)

The Java™ SE Performance Tuning

Course Objectives

• Incorporate monitoring, profiling and tuning into the application development life cycle

• Monitor the Operating System (OS) layer: Central Processing Unit (CPU), network, disk I/O, virtual memory, processes and locks

• Monitor the Java Virtual Machine (JVM) and application layers

• Profile the OS, JVM and application layers

(5)

Course Objectives

• Examine and manage the Just in Time (JIT) compiler

• Examine JVM ergonomics

• Examine 64 bit JVMs

(6)

Course Overview: What to Expect

• Master Java™ SE performance monitoring by learning:

> What and where to performance monitor

> What to profile and what tools work the best for different use cases

> Commonly observed patterns indicating performance issues

> How Java™ HotSpot garbage collectors work and how to tune them

> What you need to know about the JIT compiler

> What is JVM ergonomics and how it works

> What you need to know about 64-bit JVMs

(7)

Course Overview: What to Expect

(8)

Course Overview: The Nature of Performance Tuning

• Performance tuning is largely an art.

• There is no one approach that is always necessarily the right approach.

(9)

Course Module Outlines

● Module 1: Examining Performance Tuning

> Distinguishing between monitoring, profiling and tuning

> Incorporating monitoring, profiling and tuning into the application development life cycle

● Module 2: Monitoring the OS Layer

> Monitoring CPU utilization

> Monitoring network performance > Monitoring disk input output (I/O) > Monitoring memory utilization

(10)

Course Module Outlines

• Module 3: Monitoring the JVM and Application Layers

> Examining generational collector architectures > Monitoring GC

> Monitoring the JVM

> Monitoring the application

• Module 4: Profiling the OS, JVM and Application Layers

> Examining profiling tools > Profiling CPU usage

> Profiling the heap and memory usage > Detecting lock contention

(11)

Course Module Outlines

• Module 5: Tuning GC

> Tuning collector generation sizes

> Selecting the collector that best fits application characteristics and requirements > Examining practices that negatively impact GC performance

• Module 6: Examining and Managing the JIT compiler

> Examining choices of JIT compilers > Tuning the JIT compiler

(12)

Course Module Outlines

• Module 7: Examining Ergonomics

> Examining JVM ergonomics behavior

• Module 8: Using 64 bit JVMs

> Examine the issues associated with using 64 bit JVMs > Identify application characteristics that suit 64 bit JVMs > Tuning 64 bit JVM for different application requirements

• Module 9: Optimize the JVM for Multi-core platforms

> Examining JVM features that can leverage multi-core architectures > Optimize the JVM for various multi-core architectures

(13)

(14)

Objectives

• Distinguish between monitoring, profiling and tuning

(15)

Definitions

• Performance monitoring

• Performance profiling

(16)

Definitions: Performance Monitoring

• An act of non-intrusively collecting or observing performance data from an operating or running application.

• In most cases, a “preventative” or “proactive” type of action. However, it can be an initial step in a reactive action.

• Can be performed in production, or qualification, or development environments.

(17)

Definitions: Performance Monitoring

• Often times monitoring crosses over into trouble-shooting or service-ability.

• Training focus is on performance monitoring aspects rather than those which lend themselves to trouble-shooting or debugging.

(18)

Definitions: Performance Profiling

• An act of collecting or observing performance data from an operating or running application.

• Usually more intrusive than monitoring.

• Usually a narrower focus than monitoring.

• In general a reactive type of activity. Could be a proactive activity in situations where performance is a well defined systemic quality or requirement for a target application.

• Seldom performed in production environments.

(19)

Definitions: Performance Tuning

• An act of changing tune-ables, source code and/or configuration attribute(s) for the purposes of improving application responsiveness and/or application throughput.

(20)

Typical Development Process

Star t Analysi s Design Code Yes Quality OK Test N o

(21)

Application Performance Process

Star t Analysi s Design Code Profile N o Performance OK Benchmar k

(22)

Application Performance Process

Monitor Star t Analysi s Design Code Deploy Profile N o Yes Performance OK Benchmar k

(23)

(24)

Objectives

• Monitor CPU usage

• Monitor network I/O

• Monitor disk I/O

• Monitor virtual memory usage

(25)

What to Expect

• What level of the software stack to monitor > Operating system level

> JVM level (covered in module 3)

> Application level (covered in module 3)

• What information to monitor

> What to monitor is covered per component at a given level > Example: At OS level, monitor CPU usage

(26)

What to Monitor in the OS Level

• CPU utilization

• Network traffic

• Disk I/O

• Virtual memory usage

• Processes and kernel locks

(27)

Monitoring CPU Usage: Overview

• Rationale for monitoring CPU usage

> Get big picture view of CPU demand

> Get per process measurement of CPU utilization

• Measurements of CPU usage

> User (usr) time

> System (sys) time > Idle time

> Voluntary context switching (VCX)

(28)

CPU monitoring: What to look for

• High sys / kernel cpu time

> High sys / kernel cpu time indicates a lot of cpu cycles are spent in the kernel.

> A reduction in kernel cpu time will give more cpu time to the application.

> Also, high sys cpu time could indicate shared resource contention, (in other words, locking). More on lock contention later.

(29)

CPU monitoring: What to look for

• Idle cpu

> On multi-threaded applications and multi-core systems, idle cpu can be an indicator of an

application's inability to scale.

> Combination of high sys or kernel CPU

utilization and idle CPU could indicate shared resource contention as the scalability blocker. > Applicable to all operating systems, i.e.

(30)

Voluntary Context Switching (VCX)

bloc k bloc k Threa d Threa d Threa d

(31)

Involuntary Context Switching (VCX)

CPU Context Switch Thread Priority Interrupt Threa d Low Priority Thread Threa d

(32)

CPU monitoring: What to look for

• High VCX

> High voluntary context switching can be an indication an application is experiencing lock

contention as a result of the JVM implements or supports Java™ locking such as synchronized

methods or block, or Read/Write locks in the java.util.concurrent.locks package.

• High ICX

> There are some applications (most often seen in OLTP /

database systems) which can benefit by switching to the Solaris FX (fixed) scheduler as a means to reduce

context switching.

> For Solaris, use priocntl to set FX scheduling:

(33)

Tools For Monitoring: CPU Usage

• Tools to monitor cpu utilization > vmstat (Solaris & Linux)

> mpstat (Solaris) > prstat (Solaris)

> top (Linux, prefer prstat on Solaris) > Task Manager (Windows)

> Performance Monitor (Windows)

> Windows Resource Manager (Windows Server) > xosview (Linux)

(34)

Tools for Monitoring CPU Usage: vmstat

• Use vmstat to obtain summaries of CPU usage

(35)

CPU Usage Monitoring: vmstat

(36)

Tools For Monitoring: mpstat

• usr – user time; sys – system time; idl – idle time

(37)

(38)

CPU monitoring: prstat -m

• Example of CPU utilization microstate information measured per

process

(39)

(40)

CPU Monitoring: Solaris - cpubar

• Data of interest: 0 – CPU 0; 1 – CPU 1 avg – Average;

--- Moving average; Green – User; Red – System; Blue – Idle

• Available on Solaris Performance Tools 3.0 CD, or download from:

(41)

CPU monitoring: How to map Java™ Threads to Light Weight Processes (LWPs)

• Use HotSpot's jps command to find the process ids of all running Java™ processes on your machine.

• Use Solaris prstat -Lm, or prstat -Lmp <pid> to locate the LWP id(s) consuming the most cpu (usr or sys).

• Use HotSpot's jstack to find the executing threads taking the cpu (usr and sys) time.

• Map the LWPID to jstack's thread id.

(42)

42

Module 2: Demo 1

(43)

(44)

(45)

CPU Monitoring Using: pidstat

• %user - % of user level

(application) task

• %system - % of system level

(kernel) task

• %CPU – total % of CPU time

• cswch/s – total voluntary

context switches/second

• nvcswhc/s – total involuntary

(46)

CPU monitoring : Linux - xosview

(47)

Monitoring Network I/O: Overview

• Data of interest

> Network utilization in terms of Transaction Control Protocol (TCP) statistics and

established connections

• Tools to monitor network I/O > netstat (Solaris & Linux)

> Performance Monitor (Windows) > dtrace (Solaris)

(48)

Monitoring Network I/O: Using tcptop

• tcptop can show per process TCP statistics

• The screen capture shows 'rcp' generating 115 kb of traffic

(49)

Monitoring Network I/O: Using nicstat

• nicstat displays network statistics

(50)

Monitoring Disk I/O: Overview

> Number of disk accesses

> Latency and average latencies

• Tools to monitor disk I/O > iostat (Solaris & Linux) > iotop (Solaris & Linux) > pidstat (Linux)

> Performance Monitor (Windows) > dtrace (Solaris)

> iobar (Solaris – Performance Tools CD)

(51)

Disk I/O Monitoring Tools: iostat, iobar,

iotop, pidstat

• iostat reports per disk, text output

• iobar reports per disk, gui output

• iotop reports per process statistics, text output

• pidstat reports per process statistics, text output

> number of disk accesses, latency, average latencies

(52)

Monitoring Disk I/O: iotop example

• iotop reporting at a 5 second interval

• DISKTIME reported in microseconds

• CMD find, is keeping disk cmdk0 busy almost 60% of time during the 5 second interval

(53)

(54)

Monitoring Disk IO: Disk Cache

• Why not enable disk cache?

• What's the risk?

• On some Sun branded systems, disk cache may be disabled by default. Linux & Windows systems usually have it enabled.

• Disk cache being disabled depends on the Sun branded model and how recent the model.

(55)

Monitoring Virtual Memory: Overview

• Observe paging to identify swapping > Pages in (pi)

> Pages out (po) > Scan rate (sr)

(56)

Monitoring Virtual Memory: Swapping

• Why is swapping bad for a Java application?

(57)

Monitoring Virtual Memory: Tools

• Tools to monitor memory paging & usage > vmstat (Solaris & Linux)

> prstat (Solaris)

> top (Linux – prefer prstat on Solaris) > Performance Monitor (Windows)

> dtrace (Solaris)

> cpubar (Solaris – Performance Tools CD) > meminfo (Solaris – Performance Tools CD)

(58)

Monitoring Virtual Memory: vmstat

• Data of interest: pi – pages in; po – pages out;

(59)

Virtual Memory: Swapping Example

> pi – pages in; po – pages out; sr – page scan rate

(60)

Monitoring Virtual Memory : cpubar

> p/s - pages per second, sr - scan rate

> Watch for high scan rate, or increasing trend. Low scan rate is ok if they occur infrequently.

(61)

Virtual Memory: Fixing the

Swapping Problem

• Smaller Java heap sizes

• Add physical memory

• Reduce number of applications running on the machine

(62)

Monitoring Processes: Overview

• Data of interest > Footprint size

> Number of threads and thread state > CPU usage

> Runtime stack > Context switches > Lock contention

(63)

Monitoring Processes: Questions of

Interest

• Why is footprint size, number of threads, thread state, lock contention and context switching

important to monitor?

• What does lock contention and/or context switching look like on Solaris?

• How can you find the lock or locks causing problems?

• How can you address the thread context switching problem?

(64)

Monitoring Processes: Tools

• ps (Solaris & Linux)

• vmstat (Solaris & Linux)

• mpstat (Solaris)

• prstat (Solaris)

• pidstat (Linux)

• Performance Monitor (Windows)

• top (Linux – prefer prstat on Solaris)

(65)

Monitoring Processes: prstat -Lm

(66)

Monitoring Processes: mpstat

• Data of interest: csw – context switches; icsw – involuntary context switches; smtx – lock contention

(67)

Monitoring Processes: Kernel

> kernel cpu utilization, locks, system calls, interrupts, migrations, run queue depth

• Tools to monitor the kernel > vmstat (Linux & Solaris) > mpstat (Solaris)

> lockstat & plockstat (Solaris)

> Performance Monitor (Windows) dtrace (Solaris)

(68)

Kernel Monitoring: What to look for

• Why are high sys / kernel cpu, run queue depth, lock contention, migrations and context switching important to monitor?

> Discussion

• What do they indicate when each is observed? > Discussion

• How to do you address each of these problems? > Discussion

(69)

Kernel Monitoring Using: vmstat

(70)

Kernel Monitoring Using: mpstat

• Data of interest: Kernel cpu utilization(sys); locks(smtx);

system calls(syscl); interrupts(intr); migrations(migr); context switches (csw); involuntary context switches (iscw)

(71)

Kernel Monitoring Using : prstat -Lm

(72)

Kernel Monitoring Using: pidstat

• %user - % of user level

(application) task

• %system - % of system level

(kernel) task

• %CPU – total % of CPU time

• cswch/s – total voluntary

context switches/second

• nvcswhc/s – total involuntary

(73)

(74)

Objectives

• Examine HotSpot generational garbage collectors

• Monitor the JVM > GC

> JIT compiler

• Monitor the application > Application throughput

(75)

What to Expect

• Overview of generational GC

• How to monitoring the JVM

> GC: Which tools to use and what to monitor > JIT compiler: Which tools to use and what to

monitor

• How to monitoring the application > Application throughput

(76)

What to Monitor: JVM

• There are two major areas to monitor at the JVM level of the software stack.

> Garbage collector

>The portion of the JVM responsible for

freeing memory no longer utilized by application logic. The “magic” that lets programmers not have to worry about “managing memory”.

>Garbage collection involves traversing Java™ heap spaces where application objects are allocated and managed by the JVM's garbage collector.

> JIT compilation

>The portion of the JVM responsible for

turning byte code into execute-able

(77)

Monitoring GC

• Examining the GC Basics > Young generation

> Tenured generation > Permanent generation

> Frequency and duration of collections > Java™ heap usage

(78)

GC Basics

• HotSpot uses what is termed “generational collectors”

• HotSpot Java™ heap is allocated into generational spaces.

(79)

GC Basics: The Major Spaces

• Young generation

> Further divided into:

> Eden

>A “from” survivor space >A “to” survivor space

> Java™ objects are allocated in eden

• Tenured (old) generation

(80)

GC Basics: Young Generation

• When eden space is full, minor garbage collection event occurs. Live objects in eden space are

copied “to” a survivor space.

• Additionally, objects in the “from” survivor space are copied “to” survivor space.

• Each object which survives a garbage collection has its age incremented.

• Objects exceeding a JVM defined age threshold are promoted to the tenured (old) generation

(81)

GC Basics: Young Generation

• If “to” survivor space is too small to hold surviving Eden and “from” survivor space objects, objects will be promoted to “tenured” space.

> This is a situation which can potentially lead to performance issues.

> Short lived objects getting promoted to old

generation will require a Full GC (stop the world kind of event) or rely on a tenured spaced

concurrent collector to remove them from the Java™ heap. Both choices have their own

(82)

GC Basics: Young Generation

• Minor garbage collection events can be: > Stop the world single threaded events

Stop the world single threaded events block all Java™ application threads to perform the garbage collection event.

> Stop the world multi-threaded events

Stop the world multi-threaded events block all Java™ application threads, but is multi-threaded.

(83)

GC Basics: Young Generation

• Single threaded garbage collector is used with the “default” collector.

• Multi-threaded garbage collector, also called a parallel collector, can be used with either the “throughput” collector or “concurrent” collector.

> -XX:+UseParallelGC, throughput collector > -XX:+UseParNewGC, concurrent collector

>Throughput and concurrent collectors talked

(84)

GC Basics: Tenured (old) Generation

• Terms old and tenured generation commonly used inter-changeably

• Contains objects which have survived minor collections and copied to tenured space

> Some edge cases where objects are directly allocated

(85)

GC Basics: Tenured (old) Generation

• Objects are garbage collected by one of the following:

> Stop the world single threaded full garbage collection > Stop the world multi-threaded full garbage collection

> Stop the world means, all Java™ application threads are

blocked for the duration of the garbage collection.

> Single threaded and (mostly) concurrent garbage

collection

> Most of the garbage collection occurs concurrently while

(86)

GC Basics: Object Life Cycle

• Allocation through to garbage collection > Object is allocated in Eden space.

> If object is in-use (reachable) at minor collection time, then object is copied “to” survivor space. If object is no longer in use (not reachable) it is garbage collected.

> As subsequent minor garbage collections occur, the object is copied “from” a survivor space “to” the other survivor space.

(87)

GC Basics: Object Life Cycle

• Allocation through garbage collection (continued) > If the object continues to be in-use at

subsequent minor garbage collections, at some threshold “age”, the object will be copied

(tenured) to the tenured space. If the object is no longer in use, it will be garbage collected. > Once in the tenured space, either a concurrent

collection or full collection will be required to occur in order for the object to be collected when it is no longer in use.

(88)

88 88

(89)

GC Basics: Permanent Generation

• Contains meta-data (objects) required by the JVM to describe the objects used in the application

(90)

GC Basics: Permanent Generation

• Holds objects containing information which

describe an application class's or Java™ SE library class's methods.

• Populated by the JVM at runtime based on classes in use by the application.

• Classes may get collected (unloaded) if JVM finds they are no longer needed and space may be

(91)

(92)

Monitoring GC: The Tools

• Tools for monitoring GC > -verbose:gc > -XX:+PrintGCTimeStamps > -XX:+PrintGCDetails > -XX:+PrintGCApplicationStoppedTime > -XX:+PrintGCApplicationConcurrentTime > jstat, jps > JConsole > Java™ VisualVM > VisualGC

(93)

Monitoring GC: Using -verbose:gc

• -verbose:gc

> [GC 1884K->1299K(5056K), 0.0031820 secs]

• Use -XX:+PrintGCTimeStamps with -verbose:gc

(94)

Monitoring GC: Using -verbose:gc

• Use -XX:+PrintGCDateStamps (Java™ 6u4 and later)

> 2008-06-10T06:12:47.513-0500: [GC 10308K->2725K(101376K),

0.0320270 secs]

> Where: Date and time is GMT time.

Format: YYYY-MM-DD:HH.MM.SS.mmm-ttttt where YYYY = year, MM=month, DD=day of month, HH=hour, MM=minute, SS=seconds, mmm=milliseconds, tttt=time zone offset.

• -XX:+PrintGCDateStamps and -XX:+PrintGCTimeStamps can be

used together

> 2008-06-10T06:21:29.711-0500: 5.551:[GC

10377K->2786K(101376K), 0.0271350 secs]

(95)

Monitoring GC: Using -verbose:gc

> Frequency and duration, heap usage

• Explain what pattern(s) indicate potential problems.

(96)

96 96

(97)

Monitoring GC: Printing GC Details

• -XX:+PrintGCDetails

• [GC [DefNew: 490K->64K(960K), 0.0032800 secs] 5470K->5151K(7884K),

0.0033270 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]

• [Full GC (System) [Tenured: 5087K->5151K(6924K), 0.0971070 secs]

6047K->5151K(7884K), [Perm : 11178K->11178K(16384K)], 0.0972120 secs] [Times: user=0.10 sys=0.01, real=0.10 secs]

> frequency and duration, heap usage

(98)

98 98

(99)

Monitoring GC: Printing Pause Time

• -XX:+PrintGCApplicationStoppedTime

• -XX:+PrintGCApplicationConcurrentTime

• Helpful when tuning pause time sensitive applications

• Useful for identifying odd pause time issues when combined with GC timestamps and GC duration.

(100)

100 100

(101)

Monitoring GC: Using jps

• jps

> Included in the HotSpot JDK

> Capable of local and remote monitoring > Command line utility to find running java

processes.

> jps [-q] [-mlvV] [<hostid> where <hostid> = <hostname>[:<port>]

> Quick demo (local and remote monitoring)

(102)

Monitoring GC: Using jstat

• jstat

> included in the HotSpot JDK. > command line utility.

> jstat -<option> [-t] [-h<lines>] <vmid> [<internal> [<count>]]

> Garbage collection option(s):

>gc, gccapacity, gccause, gcnew,

gcnewcapacity, gcold, gcoldcapacity, -gcpermcapacity, -gcutil

> See jstat man page on java.sun.com for details on garbage collection options

> Explain what patterns indicate potential problems.

(103)

(104)

Monitoring GC: Using jstat

• Beware: When using the Concurrent Mark Sweep (CMS) collector (also known as concurrent

collector), jstat reports two full gc events per CMS cycle, obviously misleading. But, young

generation stats are accurate with CMS.

• Quick demo

> Running jstat against Java2D demo. > Connect locally and remote

> Remember, remote monitoring requires jstatd and a policy file.

(105)

Monitoring GC: Using jconsole

• jconsole

> Is a monitoring and management GUI console. > Is included in the HotSpot JDK.

> Can attach local or remote.

> Can monitor internals of a target JVM. > Can monitor multiple JVMs.

> Explain what patterns indicate potential problems.

(106)

Monitoring GC: Using jconsole

• Provides endless observability > MBean support for

>JVM memory usage by memory pool /

spaces

>Class loading, JIT compilation, garbage

collector, runtime, threading and logging

>Thread monitor contention

> Graphical view of heap memory, threads, cpu usage and class loading

(107)

(108)

Monitoring GC: Using VisualVM

• VisualVM: Background and capabilities > Packaged with JDK 6 update 7

> Open source project at

https://visualvm.dev.java.net

> Integrates several existing JDK software tools, lightweight memory and CPU profiling

capabilities.

>JConsole

>Subset of NetBeans Profiler

> Includes performance analysis and troubleshooting abilities.

>Thread deadlock detection >Thread monitor contention

(109)

Monitoring GC: Using VisualVM

• VisualVM: Extendability

> Can be further extended with specific

functionality for target application through:

additional in or extending an existing plug-in.

>Possibilities include:

–GlassFish performance monitoring

plug-in

–JavaDB performance monitoring plug-in –External vendors such as WebSphere

(110)

Monitoring GC: Using VisualVM

• VisualVM

> Explain what patterns indicate potential performance issues.

(111)

(112)

Monitoring GC: Using VisualGC

• VisualGC

> Standalone GUI or VisualVM plug-in. > Not included in HotSpot JDK. Separate

download.

> Visually observe garbage collection behavior. (A picture is worth a thousand words).

> Also includes classloading and JIT compilation information.

(113)

(114)

Garbage Collectors: GCHisto

• Currently standalone GUI, VisualVM plug-in under development.

• Open Source project, http://gchisto.dev.java.net

• Not included in HotSpot JDK, Separate

• Graphical tool which summarizes GC activity obtained from GC logs

• Allows comparison of JVM tuning, such as heap sizes or collector types by comparing GC logs.

(115)

Monitoring JIT Compilation

• Tools for monitoring JIT Compilation > jstat

> JConsole > VisualVM > VisualGC

> -XX:+PrintCompilation (can be intrusive) > -XX:+LogCompilation (can be intrusive)

> DTrace (HotSpot JDK 6 contains samples)

(116)

116 116

(117)

Monitoring JIT: Using PrintCompilation

• -XX:+PrintCompilation

1 java.util.Properties$LineReader::readLine (452 bytes) 2 java.lang.String::hashCode (60 bytes)

3 java.lang.String::equals (88 bytes)

3 made not entrant (2) java.lang.String::equals (88 bytes) 4 java.lang.Object::<init> (1 bytes)

5 java.lang.String::indexOf (151 bytes)

6 java.lang.String::equals (88 bytes) <--- redoing 3 6 made not entrant (2) java.lang.String::equals (88 bytes)

7 java.lang.String::indexOf (151 bytes)

8 java.lang.String::equals (88 bytes) <--- redoing 6 8 made not entrant (2) java.lang.String::equals (88 bytes)

(118)

Monitoring JIT: Using LogCompilation

• -XX:+LogCompilation

> Beware, it can be intrusive

• Will probably need someone from JIT compiler team to analyze it.

(119)

Monitoring JIT: Using .hotspot_compiler File

• What is the .hotspot_compiler file

• The .hotspot_compiler file format

> exclude A/B/C/D methodName where

>A.B.C.D is the fully qualified package and

class name and methodName is the method name.

> Example: To exclude java.util.HashMap.clear(), specify:

>exclude java/util/HashMap clear

(120)

Tools For Monitoring : JIT

• When to use the .hotspot_compiler file

> JIT compiler in an endless loop attempting a “heroic” optimization which will not converge > JIT compiler in a de-optimization –

re-optimization cycle

> JIT compiler producing 'bad' code resulting in a core dump or other severe problem

(121)

Monitoring the Application: Terms

• Throughput sensitive applications

> Have as highest priority the raw throughput of the information or data being processed

> Maximize application throughput even at the expense of responsiveness

> Will tolerate high pause times in order to maximize throughput

(122)

Monitoring the Application: Terms

• Responsiveness sensitive applications

> Have as highest priority the servicing of all requests within a predefined maximum time

> Raw throughput of data or speed of processing requests are secondary to max response time goal

> Are sensitive to GC pause time > Examples:

>User input applications such as Web browser

or GUI based applications

>Financial trading applications

(123)

Monitoring the Application: Tools and Data

• Tools

> JConsole (using application Mbeans)

> Extend VisualVM with a plug-in to gather Java™

application data of interest and monitor the application with VisualVM

> Application log

> GCHisto and btrace plugins for VisualVM > Specialized DTrace scripts

(124)

SEM-DTJ-380-LA

(125)

Objectives

• Examining and selecting profiling tools

• Examining profiling tips for: > CPU profiling

> Heap profiling

> Memory leak profiling

> Lock contention detection

• Identifying anti-patterns in: > Heap profiles

(126)

What to Expect

• What tools to use to profile

> Focus is on Sun Microsystems free and open source

tools along with bundled JDK tools

> No commercial or external vendor tools are covered

• An overview of how to use each profiling tool

• An examination of good fit use cases for each tool

• Tips for finding source of lock contention and memory leaks

• Commonly observed patterns in profiles and suggested ways to fix them

(127)

Tools For Profiling Java

™

applications

• Free tools

> NetBeans™ Profiler, subset in VisualVM

>http://www.netbeans.org

> Sun™ Studio Collector / Analyzer

> http://developers.sun.com/sunstudio/downloads > jmap / jhat

>included in HotSpot JDK • Commercial Profilers

(128)

Free Profilers: NetBeans

™

Profiler

• Characteristics:

> CPU performance profiling using byte code instrumentation

> Low overhead profiling

• Capabilities:

> Method profiling

>Select all methods for profiling or specific

method(s)

> Note: You can limit everything but JDK

classes

> Memory profiling / heap profiling > Memory leak detection

(129)

Free Profilers: NetBeans

™

Profiler

• Supported platforms:

> Solaris (SPARC & x86) > Linux

> Windows > Mac OS X

• Requirements:

> Requires HotSpot JDK 5 or later

(130)

130 130

(131)

Free Profilers: Sun Studio

• Sun Studio Collector / Analyzer capabilities: > Statistical CPU profiling using JVMTI

>Can specify sampling interval, default 1 sec >User and sys cpu time

>Inclusive or exclusive method times

> Time spent in locks

> View Java™ byte code in User Mode and Machine

Mode

(132)

Free Profilers: Sun Studio

• Sun Studio Collector / Analyzer characteristics: > Easily invoked with 'collect -j on' prefixed to

Java™ command line.

• Supported platforms:

> Solaris (SPARC & x86) and Linux

• Requirements:

> Requires HotSpot JDK 5 or later

• Additional Information:

> http://developers.sun.com/solaris/articles/perftools.html > http://developers.sun.com/solaris/articles/javapps.html

(133)

Free Profilers: Sun Studio

• View options for 'collected' data: > GUI option: Analyzer GUI

(134)

134 134

(135)

Free Profilers: jmap and jhat

• Used in combination:

> jmap – produces heap profile

> jhat – reads and presents the data

• Additional information:

> Shipped with JDK 5 and later > Command line tools

> Heap memory profiling > Perm gen statistics

(136)

136 136

(137)

Profiling Tips

• CPU profiling tips

• Heap profiling tips

• Memory leak profiling tips

• Lock contention profiling tips

• Profiling tools selection tips

(138)

CPU Profiling Tips: Why and When

• Why perform CPU profiling?

> CPU profiling provides information about where an application is spending most of its time.

• When is CPU profiling needed or beneficial?

> Poor application throughput measured against a

predetermined target

> Saturated cpu utilization

> High sys or kernel cpu utilization

> High lock contention

(139)

CPU Profiling Tips: Strategies

• Approaches which work best for CPU profiling

> Start with holistic approach to isolate major cpu consumers or hot methods.

>Look at methods with high usr and/or sys cpu

usage.

>Look at both inclusive and exclusive method

times.

>Looking at inclusive times may help identify a

(140)

CPU Profiling Tips: Strategies

• Some profilers such as NetBeans Profiler allows you profile a subset of an application.

> Approach can be useful when or if profiling the entire application is very intrusive or severely disturbs application's performance.

> If holistic approach is not possible or painful, then profiling suspected subsets of an

application is good approach.

(141)

CPU Profiling Tips: Which Product

• For profiling entire applications

> Sun Studio Collector works well

>1 second default sampling rate

>Easy to setup, just prepend 'collect -j on' to

java command line.

>Can fine tune sampling rate.

>Can direct output to specified file name.

> DTrace scripting

(142)

CPU Profiling Tips: Which Product

• For profiling portions of applications > NetBeans Profiler works very well

>Can easily configure which classes or

packages to profile, (include or !include).

>Easy to setup if application is setup as a

NetBeans Project.

>Remote or local profiling

>Can view profiling as application is running. >Can compare profile against another profile.

> DTrace scripting

>Customize to target specific portions.

>May require DTrace scripting expertise to

(143)

(144)

Heap Profiling Tips: Why and When

• Heap profiling provides information about the memory allocation footprint of an application.

• When is heap profiling needed or beneficial? > Observing frequent garbage collections > Application requires a large Java heap

> Can be useful for obtaining better cpu utilization or application throughout and responsiveness

> Less time allocating objects and/or collecting them means more

(145)

Heap Profiling Tips: Strategies

• What approaches work best for heap profiling > Start with holistic approach to isolate major

memory allocators.

>Look at objects with large amount of bytes

being allocated.

>Look at objects with high number / count of

object allocations.

>Look at stack traces for locations where large

(146)

Heap Profiling Tips: Strategies

• If holistic approach is too intrusive, NetBeans Profiler can profile subsets of the application. > Hypothesize on packages or classes which

might have a large memory allocation footprint.

> Look at objects with large amount of bytes being allocated. > Look at objects with high number / count of object allocations. > Look at stack traces for locations where large amounts of bytes

are being allocated.

> Look at stack traces for locations where large number of

(147)

Heap Profiling Tips: Strategies

• Cross reference cpu profiling with heap profiling > Look for objects which may have lengthy

initialization times and allocate large amounts of memory. They are good candidates for caching.

• Look for alternative classes, objects and possibly caching approaches where high number / count of bytes are being allocated.

• Consider profiling while application is running to observe memory allocation patterns.

(148)

Heap Profiling Tips: Complementary

Tools (jmap / jhat)

• jmap and jhat can also capture heap profiles > Not as sophisticated as NetBeans Profiler > Limited to a snapshot at the time of jmap

capture. (jmap captures the snapshot, jhat displays the data)

> User interface not as polished as NetBeans Profiler

> Easily view top memory consumer at time when snapshot was taken.

(149)

Heap Profiling Tips: jmap / jhat

Strategies

• Focus on large memory allocators

> Consider alternative classes, objects and possibly caching approaches for large

allocators.

• Capture several snapshots.

(150)

Heap Profiling Tips: jmap / jhat

Strategies

• Quick and easy to use

> run jmap on the command line > run jhat on the command line > connect with a web browser

• Can be intrusive on the application to generate the snapshot.

(151)

Memory Leak Profiling Tips: Why

• Memory leaks are situations where a reference to allocated object(s) remain unintentionally

reachable and as a result cannot be garbage collected.

• Lead to poor application performance.

• Can lead to application failure.

(152)

Memory Leaks Profiling Tips: Tools

• Tools which help find memory leaks > NetBeans Profiler

> VisualVM > jmap / jhat

> Commercial offerings (not covered)

>JProbe Memory >YourKit

(153)

Memory Leak Profiling Tips: Strategies

• NetBeans Profiler / VisualVM Strategies

> View live heap profiling results while application is running.

> Pay close attention to “Surviving Generations”.

>Surviving Generations is the number of

different object ages for a given class.

>An increasing Surviving Generations over a

period of time can be strong indicator of a source of a memory leak.

(154)

154 154

(155)

Memory Leak Profiling Tips: Strategies

• jmap / jhat Strategies

> Capture multiple heap profiles and compare footprints, (i.e. look for obvious memory usage increases).

> -XX:+HeapDumpOnOutOfMemoryError

>Use this JVM command line switch when

launching application. Can be used with -XX:HeapDumpPath=<path>/<file>

> Use jhat's Object Query Language (OQL) to query with interesting state information

(156)

156 156

(157)

Lock Contention Profiling Tips:

Overview

• Use of Java™ synchronization can lead to highly contended locks.

• Observing high values of voluntary context

switches can be an indication of lock contention.

• Collector / Analyzer is very good with identifying Java objects experiencing lock contention.

(158)

158 158

(159)

Profiling Tips: Good Use Cases

• Collector / Analyzer

> CPU profiling entire application

> Sys cpu profiling or distinct usr vs sys profiling > Lock contention profiling

> Integration with scripts, command files or batch files

> Also view performance of JVM internals including methods

(160)

Profiling Tips: Good Use Cases

• NetBeans Profiler

> Profiling subset of application, for CPU profiling or heap profiling

> Heap profiling

> Finding memory leaks

> Profiling an application using NetBeans IDE and/or NetBeans project

> Remote profiling

> Attach to running application

(161)

Profiling Tips: Good Use Cases

• DTrace and DTrace scripts

> Non-intrusive snapshots of running application > Command line utility

> Can leverage existing public scripts

>Heap profiling

>Finding memory leaks >Monitor contention

>JIT Compilation

(162)

Profiling Tips: Good Use Cases

• jmap / jhat

> Heap profiling

> Finding memory leaks

> Simple command line utilities

(163)

Profiling Tips: Inlining effect

• If observing misleading or confusing results in cpu profiles, disable in-lining

• It is possible methods of particular interest are being in-lined and leading to misleading

observations.

• To disable in-lining, add the following JVM

command line switch to the JVM command line args: -XX:-Inline

(164)

Identifying Anti-patterns and

Memory Leak Patterns

• Identifying anti-patterns in heap profiles

• Identifying memory leak patterns in heap profiles

(165)

Anti-patterns in Heap Profile

• Large number of String or char[] allocations in heap profile

> Possible over allocation of String

> Possibly benefit from use of StringBuilder

> Possible StringBuilder or StringBuffer resizing. > Possibly utilize ThreadLocal to cache char[] or

StringBuilder or StringBuffer

• Reducing char[] and String allocations will likely reduce garbage collection frequency.

(166)

Anti-patterns in Heap Profiles

• Observing StringBuffer in heap profile > Possible candidate for StringBuilder if

synchronized access is not required. > Reducing char[] allocations on

StringBuilder/StringBuffer on expansion of StringBuilder/StringBuffer size.

• Reducing char[] allocations will likely reduce garbage collection frequency.

(167)

Anti-patterns in Heap Profiles

• Observing Hashtable in heap profile > Possible candidate for HashMap if

synchronized access is not required.

> Possible candidate for ConcurrentHashMap if synchronized access is required.

> Further partitioning of data stored in Hashtable may lead to finer grained synchronized access and less contention.