Methodology

In order to monitor runtime-compiled code, our strategy is to first start tracking the processes which are expected to generate executable code at runtime. When the target bytecode is intended to be JIT compiled, or executable code is prepared in a process, a small code cache is prepared in the memory where the generated machine code is saved. This code cache may also be updated repeatedly with the updated CSrsections, as seen in Figure 7.2. Our approach

keeps track of each update for each page change of the code-cache. On a POSIX-compliant system, the malloc() syscall is used to allocate memory for this cache and its attributes flags are set to executable, using a mprotect() syscall and the PROT_EXEC flags supplied with it. We essentially keep track of access rights on this memory section and generate artificial page- faults from the kernel to extract the executable pages. The following subsection describes the FlowJIT approach in detail.

7.4.1 FlowJIT Architecure

FlowJIT begins by tracking the individual pages in the kernel for the tracked process which is runtime compiling native code for execution. As shown in Figure 7.3, a userspace program flowjit-agent is used to register the target process using a FlowJIT specific ioctl(). This sets a flag in the kernel task structure for the process, which then marks the process to

be monitored. Next, FlowJIT intercepts the access protection functions in the kernel which

Runtime Code Userspace

Kernel

Page Access Control Target Process Runtime Code Registration Control Tracked Pages ioctl() flowjit-agent NX NX PF Handler X X Trace Decoder ID Timestamp Instruction Pointer Runtime Code Runtime Code Query FlowJIT Events

Figure 7.3 FlowJIT Architecture

identify the executable pages requested by the target process. As discussed before, these pages correspond to the CSr code sections. The page access control module in FlowJIT sets those

pages as tracked and flips their protection rights from executable to non-executable (NX). The Page Fault Handler manages access-faults on those pages and sets the access rights back to executable. At the same time, it copies the executable code from the kernel and generates a synthetic FlowJIT event by adding the current time-stamp, the faulting instruction IP, and assigning an ID. Any update to the JIT code for the same IP is stored in a linked list in the kernel. A userspace daemon, part of the hardware trace decoder framework, lazily copies the linked list contents from the exposed DebugFS file to the disk. The decoder can then query the FlowJIT events based on their IP and synchronize them with the timestamp to complete the code reconstruction. The linked list data is indexed according to the virtual address for the dynamic pages and queried by address and time when the decoder encounters a virtual address where the decoding failed. We demonstrate this in detail in section 7.5, where we

properly reconstruct the otherwise incompletely decoded hardware trace gathered from the Intel PT trace hardware.

In-Kernel Access Tracking

The access management part of the FlowJIT system can be defined using a state machine where the states denote the status and access rights of pages in a virtual memory area (VMA). In our context, the pages may be in state either Tracked, Untracked or Fault. In Figure 7.4, we show how these states can transition, and the access rights on the pages thereafter. Initially,

Untracked (EXEC/NON-EXEC) EXEC→NON-EXEC Tracked (EXEC) Tracked (NON-EXEC) TRACK TRACK Fault (NON-EXEC→EXEC, DUMP) UN-TRACK UN-TRACK

Figure 7.4 Access Tracking State Machine

when the process registers itself with the kernel using the flowjit-agent, all its pages, whether executable or non-executable, are marked as tracked and remain in this state until the agent requests them to be untracked. Tracked (Exec) pages which are executable are artificially marked as non-executable by FlowJIT and they enter the Tracked (Non-Exec) state. In the normal course of execution, when the application tries to execute them, they generate a page fault and enter the Fault state. We handle that page fault in the kernel and query if the page was originally tracked and its attributes changed. In such a case, the page attributes are changed back to their original values and a synthetic FlowJIT event is generated. The event is dumped and the page again enters the Tracked (Exec) state and is then untracked by the agent. The underlying process has been illustrated in Algorithm 3

Algorithm 3 Access Tracking Algorithm Input : VMAs containing CSr and CSp for P

Output : Updated FlowJIT event buffer (Bev) 1: Vr ← vma(CSr)

2: Vp ← vma(CSp) 3: V = {Vr, Vp}

4: while P.pid is tracked and page in exec_pages(V ) do

5: if (page ∈ Vr) then

6: page.arm = armed

7: _{FlowjitToggle(page, INIT)} 8: end if

9: if (page.fault_exec) then

10: _{FlowjitHandle(page)} . Access-fault handler

11: end if

12: end while

13: _{function FlowjitToggle(page, state)} 14: if (state is INIT) then

15: page.exec ← 0 . FlowJIT sets NX

16: else if (state is PF) then

17: ev ← {timestamp, address(Vpage)} 18: _{FlowjitEventDump(ev)}

19: page.exec ← 1 . FlowJIT resets NX

20: end if

21: end function

22: _{function FlowjitHandle(page)} 23: if (page.arm is armed) then

24: page.arm = disarm 25: _{FlowjitToggle(page, PF)} 26: end if 27: end function 28: _{function FlowjitEventDump(event)} 29: Bev ← event

30: f lush(Bev) . Buffer copied to disk

31: end function

We have released FlowJIT as an open-source kernel patch1_{which can be activated by a simple}

configuration option in the kernel. While most of the other runtime code profiling techniques require compiler-specific APIs or manual addition of code in the target application/runtime compiler engine, the FlowJIT technique requires no other modification to the source code and tackles the problem of runtime code reconstruction at kernel level. This makes it versatile to use as well as the first approach to tackle runtime compiled code reconstruction for hardware

traces. We explain this using two illustrative examples in the next section.