Production time profiling On-Demand with Java Flight Recorder

(1)

(2)

(3)

Production time profiling

On-Demand

with Java Flight Recorder

Klara Ward

Principal Software Developer

Java Platform Group, Oracle

Oracle Confidential –

Internal/Restricted/Highly

(4)

About me

–

Developer in the Oracle Java Mission Control team

in Stockholm, Sweden

–

Sometimes tweets @klaraward

(5)

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for

information purposes only, and may not be incorporated into any contract. It is not a

commitment to deliver any material, code, or functionality, and should not be relied upon

in making purchasing decisions. The development, release, and timing of any features or

functionality described for Oracle’s products remains at the sole discretion of Oracle.

(6)

Agenda

Overview of Java Mission Control

Overview of Java Flight Recorder

Demo

Customization, Future, Links

Q&A

1

2

3

4

5

6

(7)

“I managed to do in one day what I've failed

to do in 2+ weeks using <profiling tool> and

<another profiling tool>.”

“ JMC is my main tool for getting insight into

the rhythm of a JVM and the running

applications. … I have used recordings to

resolve critical production issues caused by

latency, memory-leaks or threading. ”

(8)

RebelLabs Developer Productivity Report 2015,

Java Performance Survey

(9)

”Java Mission Control profiling tool”

Probably:

–

Data from Java Flight Recorder

–

Visualized in Java Mission Control

(10)

Java Flight Recorder (JFR) & Java Mission Control (JMC)

Brief overview

JMC

JDK

JVM

Low

overhead

JFR

Engine

JFR

Events

Recording data

myrecording.jfr

Control recordings

start/stop/dump

java -XX:+FlightRecorder

JDK/bin/jmc

or Eclipse plug-ins

JDK/bin/jcmd <pid> <cmd>

(11)

Overview of Java Mission Control

The graphical client

(12)

Java Mission Control Overview

• A tools suite for

production

use (fine in development too)

–

Basic monitoring

–

Production time

profiling

and diagnostics

• Free for development and evaluation

–

Tool usage is free, data creation in production

requires a commercial license

(13)

History of Mission Control

• JRockit Flight Recorder

• _{Appeal(JRockit) -> BEA Systems ->}

_Oracle

_{<- Sun}

• _{Best JRockit features -> HotSpot JVM}

(14)

Java Mission Control Main Tools

Two main tools:

• JMX Console

–

Online monitoring

• Flight Recorder

–

Offline low overhead profiler

–

Control and visualization in JMC

(15)

Experimental Plugins

• DTrace

–

JFR style visualization of data produced by DTrace

• _JOverflow

–

Memory anti-pattern analysis from hprof dumps

• JMX Console plug-ins

• Java Flight Recorder plug-ins

–

WLS

–

JavaFX

(16)

JMC installation/startup

<JDK>/bin/jmc

–

Mac:

(/usr/bin/) jmc

Add if needed:

–consoleLog –debug ( | more 2>&1 )

• Eclipse plug-ins

–

Install from update site on OTN:

http://oracle.com/missioncontrol

, Eclipse Update Site

• Experimental plug-ins: Install from within the JMC app, or from

(17)

Overview of Java Flight Recorder

Low overhead profiling

(18)

Flight Recorder

• High Performance Event Recorder

• Built into the JVM

–

Already available runtime information

–

Measuring the real behavior, doesn’t disable JVM

optimizations

• Binary recordings

–

Self contained self describing chunks

• Very detailed information

• _{Extremely low overhead (~1-2%)}

–

Can keep it always on, dump when necessary

(19)

Java Flight Recorder (JFR) & Java Mission Control (JMC)

Brief overview

JMC

JDK

JVM

Events

JFR

Engine

Recording data

myrecording.jfr

Control recordings

start/stop/dump

java -XX:+FlightRecorder

JDK/bin/jmc

or Eclipse plug-ins

JDK/bin/jcmd <pid> <cmd>

java -XX:StartFlightRecording

JFR Java

API

JFR JMX

API

Java

Events

(20)

Data collected by JFR

• Java application behavior

–

Threads/Locks

–

I/O

–

Exceptions

• JVM behavior (indirect Java application behavior)

–

Garbage collection, allocation

–

JIT Compiler

(21)

Method sampling

• Sampling profiler

–

Not displaying every single call to your method

• This is partly why we get the low overhead

• Detects hot methods

–

Does not require threads to be at safepoints

• (Flags currently needed to give more accurate non-safepoint data)

-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints

(will be default in coming releases)

(22)

RebelLabs Developer Productivity Report 2015,

Java Performance Survey

(23)

RebelLabs Developer Productivity Report 2015,

Java Performance Survey

(24)

Different Kinds of Events

• Instant Event -

Exception

• Duration Event –

Thread.sleep

–

Configurable

threshold

• Requestable Event –

Method profiling sample

–

Polled from separate thread

–

Configurable

period

(25)

Event settings

• Predefined settings

–

‘default’ – designed to get max information within <= 1 % overhead

–

‘profile’ – even more information, ~2 % overhead

–

Enabling of event types, configuring periods and thresholds

–

jre/lib/jfr/*.jfc

(26)

Different Kinds of Recordings

• Continuous Recordings

–

Have no end time

–

Must be explicitly dumped

–

Example use case: Enable at startup, dump the last X minutes when needed

• _{Time Fixed Recordings (‘profiling recordings’)}

–

Have a fixed time

–

If started from Java Mission Control, opened automatically in the GUI

–

Example use case: Performance testing under load, do a 1 minute recording

(27)

Creating recordings

More than one way

(28)

Preparations

• Start the JVM from which to get recordings with:

-XX:+UnlockCommercialFeatures

-XX:+FlightRecorder

• In 8u40 and later, possible to enable at runtime if needed

–

Using JMC or jcmd

–

“On-demand”

(29)

Creating Recordings Using Mission Control

1. Find a JVM to do a recording on in the JVM Browser

2. Double click the Flight Recorder node under the JVM

3. Follow the wizard

NEW: No need for the JVM flags,

automatic enablement from JMC

(30)

Creating Recordings Using Startup Flags

-XX:+UnlockCommercialFeatures -XX:+FlightRecorder

• Time fixed

–

-XX:StartFlightRecording= delay=20s,duration=60s,

filename=C:\tmp\myrecording.jfr,settings=profile,name=JavaLand

• _{Continuous w/ dumponexit}

–

-XX:StartFlightRecording=settings=default

-XX:FlightRecorderOptions=dumponexit=true,

dumponexitpath=C:\tmp\myrecordings

–

(Needed before 8u20:

-XX:FlightRecorderOptions=defaultrecording=true )

(31)

Creating Recordings Using JCMD

Usage:

jcmd <pid> <command>

• Starting a recording:

jcmd 7060 JFR.start name=MyRecording settings=profile

delay=20s duration=2m filename=c:\tmp\myrecording.jfr

• Dumping a recording:

jcmd 7060 JFR.dump name=MyRecording

filename=C:\tmp\dump.jfr

• Unlocking commercial features (if JVM not started with the flag):

jcmd 7060 VM.unlock_commercial_features

Don’t forget to try

jcmd <pid> help

to see what else jcmd can do

(32)

Creating Recordings Using JMX Console triggers

• Start JMC

• Connect a JMX Console to your

application

• Configure and enable rules on the

Triggers tab

• Recording will be started or dumped

when the trigger occurs

(33)

Remote production systems

• Remote access

–

either using

-Dcom.sun.management.jmxremote

…, connect with JMC

–

or start recording from local commandline using

jcmd

, transfer JFR to workstation

• Enabling JFR

–

at startup

• some very small initiation overhead at startup, threads allocate small amount of

extra memory

–

dynamically

• 0% overhead at startup, initiation overhead happens at time of enabling, might

cause some classes to be deoptimized etc.

• If you want to avoid restart

(34)

Analyzing recordings

Using the graphical client

(35)

How to think about the information shown

• _{Only you know what your application is supposed to be doing}

–

Batch job, or real time trading?

–

Do you want the CPU usage to be high or low?

–

If you have a theory about what is wrong, you can find out why

(36)

How to think about the information shown

• CPU load

–

Low load – method sampling not very interesting

–

Full load – latencies not very interesting/likely

• Event thresholds

–

Keeping performance up but still detecting outliers

–

Is the thread running normally during 30%?

–

Hint: Default threshold is 10 ms.

(37)

DEMO!

Flight Recorder startup and analysis

(38)

Customization

38

(39)

import com.oracle.jrockit.jfr.*;

public

class

Example {

private

final static String PRODUCER_URI =

"http://www.example.com/demo/"

;

private

Producer

myProducer

;

private

EventToken

myToken

;

public

Example()

throws

URISyntaxException, InvalidEventDefinitionException, InvalidValueException {

myProducer

=

new

Producer(

"Demo Producer"

,

"A demo event producer."

, PRODUCER_URI);

myToken

=

myProducer

.addEvent(MyEvent.

class

);

}

@EventDefinition(path=

"demo/myevent"

, name =

"My Event"

,

description=

"An event triggered by doStuff."

, stacktrace=

true

, thread=

true

)

private

class

MyEvent

extends

TimedEvent {

@ValueDefinition(name=

"Message"

, description=

"The logged important stuff."

)

private

String

text

;

public

MyEvent(EventToken eventToken) {

super

(eventToken); }

public

void

setText(String

text

) {

this

.

text

=

text

; }

}

public

void

doStuff() {

MyEvent

event

=

new

MyEvent(

myToken

);

event

.begin();

String

importantResultInStuff

=

""

;

// Generate the string, then set it...

event

.setText(importantResultInStuff);

event

.end();

event

.commit();

}

(40)

...

private

MyEvent

event

=

new

MyEvent(

myToken

);

public

void

doStuffReuse() {

event.reset();

event.begin();

String

importantResultInStuff

= "";

// Generate the string, then set it...

event.setText(

importantResultInStuff

);

event.end();

event.commit();

}

...

• NB: If JFR is not enabled, the custom events gives no (0%) overhead

Adding Your Own Events

(unsupported)

(41)

• The JDK parser

• import

oracle.jrockit.jfr.parser.*;

• SAX style parser

• The JMC parser

• import

com.jrockit.mc.flightrecorder.FlightRecording;

• DOM style parser

• http://hirt.se/blog/?p=446

Parsing recordings

(unsupported)

(42)

Built in GUI editor

(unsupported)

• Show

view -> Designer

• Customize the existing GUI or

produce entirely new GUIs

for events

• Export the created GUI to

share it with others

(43)

Future

(44)

Future

• JFR -

Supported

API for adding your own JFR events

• JMC - Automatic analysis of Flight Recordings

–

”You have used a JVM flag that is not recommended”

–

”Your application is doing a lot of GC, investigate more here”

(45)

Resources

Homepage:

http://oracle.com/missioncontrol

(Click Discussion to see the forum)

http://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/index.html

Twitter:

@javamissionctrl

@klaraward

@hirt

Blog:

http://hirt.se/blog?

(

http://hirt.se/blog/?p=611

for JMC tutorial)

Facebook:

(46)

Questions?

Q: Can you repeat those JVM flags again?

(47)

Safe Harbor Statement

The preceding is intended to outline our general product direction. It is intended for

information purposes only, and may not be incorporated into any contract. It is not a

commitment to deliver any material, code, or functionality, and should not be relied upon

in making purchasing decisions. The development, release, and timing of any features or

functionality described for Oracle’s products remains at the sole discretion of Oracle.

(48)

(49)