<Insert Picture Here> Tracing on Linux: the Old, the New, and the Ugly

<Insert Picture Here>

Categories of Tracing Tools

•

_{Kernel Tracing}

•

_{Userspace Applications Tracing}

•

_{Static Tracing}

•

_{Dynamic Tracing}

•

_Sampling

•

_Continuous

A Brief History

• _{LTT (Linux Trace Toolkit) (1998) replaced in 2006} • _{Kprobes (2004):}

http://www.ibm.com/developerworks/library/l-kprobes/index.html

• _{Dtrace for Solaris (2003) released on Solaris 10 in November 2005} • _{Systemtap for Linux (2005) at OLS}

• _{LTTng (LTT Next Generation) (2006): https://lkml.org/lkml/2006/1/9/223} • _{Ftrace: (2008): https://lkml.org/lkml/2008/1/3/26}

Currently Active Projects

•

_Ftrace

•

_Perf

•

_Systemtap

Ftrace

• _{In kernel with portions outside}

• _{Main Maintainer: Steven Rostedt (2008)}

• _{CLI: /sys/kernel/debug/tracing as interface (control and output)} • _{GUI for data visualization (KernelShark)}

http://people.redhat.com/srostedt/kernelshark/HTML/

• _{Trace-cmd: user space tool with subcommands}

http://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git

• _{Fundamental mechanism: gprof hooks. Adds mcount() call at entry of each}

kernel function. Requires kernel to be compiled with -pg option for some functionality

• _{User: Root privileges} • _{No user space tracing}

Information collected: ftrace plugins

• _{Function: trace entry of all kernel functions}

• _{Function-graph: traces on both entry and exit of the functions. It}

then provides the ability to draw a graph of function calls

• _{Wakeup: max latency that it takes for the highest priority task to}

get scheduled after it has been woken up

• _{Irqsoff: areas that disable interrupts and saves the trace with the}

longest max latency

• _{Preemptoff: amount of time for which preemption is disabled} • _{Preemptirqsoff: largest time for which irqs and/or preemption is}

disabled

• _{Nop: trace nothing}

Ftrace: trace-cmd

• _{User space tool} • _{Requires root}

• _{Some commands are:}

• Record: start collection of events into file trace.dat

• _{Can use ftrace plugins (-p) or static kernel events (-e)}

• Report: display content of trace.dat

• Start: start tracing but keep data in kernel ring buffer

• Stop: stop collecting data into the kernel ring buffer

• Extract: extract data from ring buffer into trace.dat

• List: list available plugins (-p) or events (-e)

• _{Many options, very flexible}

Ftrace: Original Interface

Ftrace: Trace-cmd

Ftrace: Dynamic Tracing

• _{Added by Masami Hiramatsu (end of 2009)} • _{Real dynamic tracing}

• _{Use /sys/kernel/debug/tracing/kprobe_events to control from command}

line

• _{Flexible to an extent, but hard to use because of syntax} • _{Read more: Documentation/trace/kprobetrace.txt}

• _{LWN article: http://lwn.net/Articles/343766/}

echo 'r:myretprobe do_sys_open $retval' >  /sys/kernel/debug/tracing/kprobe_events

Perf

• _{In kernel (tools/perf directory) userspace tool (started in 2008)} • _{Main contributors : Thomas Gleixner, Ingo Molnar, Frederic}

Weisebecker, Arnaldo Charvalo De Melo, Peter Zijlstra and others

• _{Started as Hardware performance counters interface, initially called}

perf counters.

• _{Originally on top of the hardware-based performance monitoring}

interface for Linux by Stephane Eranian. Now available as libpfm version 4 http://perfmon2.sourceforge.net/docs_v4.html

• _{Has grown into all encompassing tracing system} • _{Very active at the moment}

• _{No user space tracing yet}

• _{Documentation: tools/perf/design.txt and tools/perf/Documenation} • _{Perf wiki: https://perf.wiki.kernel.org/index.php/Main_Page}

Perf subcommands

• _{Perf stat: collects and display events data during a command}

execution

• _{Perf record: run a command, store its profiling (sampling mode)}

in output file (perf.data)

• _{Perf report: display data previously recorded in output file}

(perf.data)

• _{Perf diff: diff between two perf.data files}

• _{Perf top: performance counter profile in real time}

• _{Perf probe: define dynamic tracepoints (more on this later)}

• _{Perf kmem: show statistics on kernel mem (from perf.data), or}

record kernel mem usage for a command

• _{Perf trace-perl (trace-python): process previously recorded trace}

Perf subcommands (continued)

• _{Perf list: list available symbolic events types (in exadecimal}

encoding)

• _{Perf annotate: display perf.data with assembly or source code of}

the command executed

• _{Perf lock: lock statistics}

• _{Perf sched: scheduler behaviour}

• _{Perf kvm: perform same as above but on a kvm guest}

environment with subcommands:

• Perf kvm top

• Perf kvm record

• Perf kvm report

Perf: Types of events

• _{Perf list shows the list of predefined events that are available} • _{Use one or more of the events in perf stat commands:}

perf stat -e instructions sleep 5

• _{Hardware: cpu cycles, instructions, cache references, cache misses, branch}

instructions, branch misses, bus cycles. Uses libpfm. CPU model specific.

• _{Hardware cache: for instance L1-dcache-loads, L1-icache-prefetches} • _{Hardware breakpoint}

• _{Raw: hexadecimal encoding provided by harware vendor}

• _{Software: in kernel events. Such as: cpu clock, task clock, page faults, context}

switches, cpu migrations, page faults, alignment faults, emulation faults.

• _{Static tracepoints: needs the ftrace event tracer enabled. Events are listed in}

/sys/kernel/debug/tracing/events/* (locations defined with TRACE_EVENT() macros)

Other Control Parameters

• _{Process specific: default mode, perf follows fork() and}

pthread_create() calls

• _{Thread specific: use - - no-inherit (or -i) option} • _{System wide: use -a option}

• _{CPU specific: use -C with list of CPU numbers} • _{For running process: -p <pid>}

• _{To find what binaries were running , from a perf.data file, use Elf}

unique identifier inserted by linker:

• perf buildid-list

• _{User, Kernel, Hypervisor modes: count events when the CPU is}

in user, kernel and/or hypervisor mode only. Use with perf stat

Dynamic Tracing in Perf

• _{Contributed by Masami Hiramatsu}

• _{Probes can be inserted on the fly using perf probe} • _{Defined on the command line only}

• _{Syntax can be simple or complex}

• _{Can use dwarf debuginfo to show source code (--line option)}

before setting probe (uses elfutils libdw like systemtap), or to use function name, line number, variable name

• _{Abstraction on ftrace kprobes, alleviates usage problems,}

supports also same syntax to specify probes

• _{Some options: --add, --del, --line, --list, --dry-run, --verbose,}

--force

Example

Add probe, run a command, and display the data:

# ./perf probe -P 'p:myprobe @fs/read_write.c:285 file buf count'

Adding new event: p:myprobe vfs_read+57 file=%bx buf=%si count=%ax # ./perf list

...

rNNN [raw hardware event descriptor] kprobes:myprobe [Tracepoint event]

# ./perf record -f -e kprobes:myprobe:record -F 1 -a ls #./perf trace

[...]

perf-11445 [000] 95862.048894383: myprobe: (c04bbed5) file=dae15e80 buf=b78b2000 count=400 perf-11445 [000] 95862.049066533: myprobe: (c04bbed5) file=dae15d80 buf=b78b2000 count=400 [...]

LTTng

•

_{External to kernel tree}

•

_{Maintainer Mathieu Desnoyers}

•

_{http://lttng.org/}

•

_{Userspace tracing via markers and tracepoints (static}

instrumentation). Need to link with special library.

•

_{kprobes/tracepoints support for kernel tracing}

•

_{GUI to view events (LTTV)}

•

_{Eclipse integration}

•

_{Included in some embedded Linux distributions}

Systemtap

• _{External to kernel tree (started in 2005)} • _{Multiple maintainers: Red Hat, IBM, others} • _{http://sourceware.org/systemtap/wiki}

• _{Kernel tracing using kprobes}

• _{Userspace tracing requires in kernel infrastructure that is not in}

mainline (only Red Hat based kernels) (utrace/uprobes)

• _{Dynamic probes and tracepoints}

• _{Has well defined rich scripting language} • _{GUI available}

• _{Safety issues around modifying memory locations (guru mode)} • _{Can use debuginfo (seen as drawback)}

Kprobes: Dynamic Tracing

•

_{Added to the Linux kernel in 2005}

•

_{Similar to breakpoints: place breakpoint instruction at}

desired code location and execute actions when hit.

•

_{Optimizations to kprobes using Jumps instead of}

exceptions.

Events Markers

• _{Static probe points in kernel code}

• _{Independent of users (ftrace, perf, Lttng, systemap)} • _{Many uses in the kernel (a few hundreds and growing)} • _{TRACE_EVENT() macro with 6 arguments}

• _{Definitions in include/linux/tracepoint.h}

• _{Define characteristics of tracing actions (probe) using}

TRACE_EVENT() in include/trace/events/*.h

• _{Mark tracing locations with function call trace_my_event_name()}

• e.g. trace_sched_switch() in sched.c and

TRACE_EVENT(sched_switch,...) defined in sched.h

• _{Read the 3-article series:}

• http://lwn.net/Articles/379903/

Utrace

• _{Systemtap provides userspace tracing based on utrace. Code in}

RHEL/Fedora.

• _{Utrace: goal was to be replacement for ptrace. Status: development}

abandoned. Fedora still ships it, used by a few other tools.

• See mailing list: https://www.redhat.com/mailman/listinfo/utrace-devel

• See code: http://redhat.com/~roland/utrace/

• _{Maintained by Oleg Nesterov}

• _{Patch kept up to date with recent kernels, most recent is for kernel 3.1}

Uprobes

• _{Uprobes: handle userspace breakpoints in kernel} • _{Analogous to kprobes}

• _{Uses breakpoint instruction} • _{Posted many revisions} • _{No longer based on utrace} • _{No signals}

• _{Multiple tracers allowed}

• _{Ptrace replacement eventually?}

• _{Still not approved}

• _{Latest version of the patchset https://lkml.org/lkml/2011/9/20/123}

• _{Integration of previous version of the patch with latest kernel F15 and F16}

Tracing Infrastructure Unification?

• _{Goal: unify the kernel infrastructure for tracing}

• _{There are separate ring buffers used by perf and ftrace}

• Need more complex interfaces

• Need to be fast

• _{Simpler ring buffers in the kernel (kfifo) for other users also exist} • _{Unification effort restarted many times:}

http://lkml.org/lkml/2010/7/9/368

http://lkml.org/lkml/2010/8/17/485

• _{Some old benchmarks:}

http://git.kernel.org/pub/scm/linux/kernel/git/compudj/linux-2.6-ringbuffer.git

• _{Status quo with multiple buffers is likely to persist. Too many differing needs.} • _{Possible unification of ftrace and perf? Proposal of “trace” but stalled:}

Dtrace

• _{A Solaris tool, available since 2005}

• _{Ported to Linux by Oracle, initial release October 2011, version 0.1, still WIP} • _{Code posted here http://oss.oracle.com/git/}

• linux-2.6-dtrace-modules-beta.git

• linux-2.6-dtrace-unbreakable-beta.git

• _{Dtrace provider, syscall provider}

• _{Next: static probes, then tackle userspace tracing}

• _{Integrated with Oracle Unbreakable Enterprise Kernel, available on UEK}

2.6.32 right now (0.1 version), moving to UEK2 version 2.6.39 from now on

• _{x86_64 only}

• _{Kernel changes are GPL} • _{Kernel Module is CDDL}

In Summary

• _{Systemtap: Moving forward, but on its own. Provides userspace tracing.}

Resistance by kernel community around architectural design decisions. Integration as is with upstream kernel unlikely?

• _{LTTng: Developed actively, but standalone. Provides userspace tracing. Code in}

its own repo. Not in mainline kernel.

• _{Ftrace: continuing, thanks to trace-cmd and kernelshark. Kernel focused} • _{Perf: most active project at this time. Kernel focused}

• _{Unification of Buffers in kernel: stalled}

• _{Scalability: in progress. See google's comments: http://lwn.net/Articles/464268/} • _{Utrace: development stalled/abandoned, maintenance only}

• _{Uprobes: hopeful}

• _{In kernel interpreter: maybe?}

• _{tracepoints/static events: KABI status?} • _{Integration of ftrace and perf? trace?}

Thank You

•

_{Ananth Mavinakayanahalli (IBM)}

•

_{Srikar Dronamraju (IBM)}

•

_{Masami Hiramatsu (Hitachi)}

•

_{Steven Rostedt (Red Hat)}

The preceding is intended to outline our general

product direction. It is intended for information

purposes only, and may not be incorporated into any

contract. It is not a commitment to deliver any

material, code, or functionality, and should not be

relied upon in making purchasing decisions.

The development, release, and timing of any

features or functionality described for Oracle’s

products remains at the sole discretion of Oracle.