Cloud9 Parallel Symbolic Execution for Automated Real-World Software Testing

(1)

Parallel Symbolic Execution for

Automated Real-World Software Testing

Stefan Bucur, Vlad Ureche, Cristian Zamfir, George Candea

Cloud9

(2)

Automated

Techniques

Automated Software Testing

λ

Symbolic Execution Model Checking

Industrial

SW Testing

Manual Testing

Static Analysis

Fuzzing

Scalability

Applicability

Usability

(3)

Cloud9 - The Big Picture

• Parallel symbolic execution

• Linear scalability on commodity clusters

• Full symbolic POSIX support

• Applicable on real-world systems

• Platform for writing test cases

(4)

Automated Systems Testing

[*] C. Cadar, D. Dunbar, D. Engler, “KLEE: Unassisted and automatic generation

• Promising for systems testing:

KLEE

[*]

• High-coverage test cases

• Found new bugs

• ... But applied only on small

programs

λ

(5)

Memcached

GNU Coreutils

Apache

(6)

void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

(7)

Symbolic Execution in a Nutshell

(8)

pkt->magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

(9)

pkt->cmd == GET pkt->magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

(10)

pkt->cmd == GET pkt->magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

(11)

Symbolic Execution in a Nutshell

(12)

λ.magic == 0xC9 _{λ.magic != 0xC9} void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

(13)

λ.cmd == GET λ.cmd != GET λ.magic == 0xC9 _{λ.magic != 0xC9} void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

(14)

λ.cmd == GET λ.cmd != GET λ.magic == 0xC9 _{λ.magic != 0xC9} void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... }

Symbolic Execution in a Nutshell

∼

2 paths

λ

program

size

(15)

CPU Bottleneck

(16)

W1

_W2

W3

(17)

W1

W2

W3

Parallel Tree Exploration

Key research problem:

(18)

Linear Solution to Exponential

Problem

Program Size

Time t

o

Test

(19)

Linear Solution to Exponential

Problem

Program Size

Time t

o

Test

Testing target 1 worker

(20)

Linear Solution to Exponential

Problem

Program Size

Time t

o

Test

Testing target

Bring testing time down to practical values

1 worker

2 workers

4 workers 8 workers

(21)

(22)

Scalability Challenges

Tree structure not known a priori

?

_?

(23)

Scalability Challenges

(24)

(25)

Scalability Challenges

(26)

(27)

Outline

• Scalable Parallel Symbolic Execution

• POSIX Environment Model

(28)

Cloud9 Architecture

Global

(29)

Cloud9 Architecture

W1’s Local Tree

W2’s Local Tree

W3’s Local Tree

Each worker runs a local

(30)

Cloud9 Architecture

Candidate

nodes

Fence

nodes

• Candidate nodes are selected for

exploration

(31)

Load Balancing

LB

W1

W2

W3

(32)

Load Balancing

LB

W1

W2

W3

(33)

Load Balancing

LB

W1

W2

W3

(34)

Work Transfer

W1

(35)

Work Transfer

(36)

Work Transfer

W1

W2

Virtual

(37)

Work Transfer

W1

W2

(38)

Work Transfer

W1

W2

Materialized

(39)

Work Transfer

W1

W2

(40)

1 1 1 1 1 0 0 0 0 0 0 0 0

Path-based Encoding

• Nodes are encoded as paths in tree

• Compact binary representation

• Two paths can share common prefix

• Small encoding size

• For a tree of

2

100

_{leaves, a path fits in}

(41)

Load Balancing in Practice

LB stops after 1 min LB stops after 4 min Continuous load balancing

W o rk d o n e [% o f to ta l in st ru ct io n s] Time [minutes]

0

10

20

30

40

50

60

70

80

90

100

0

2

4

6

8

10

(42)

Outline

• Scalable Parallel Symbolic Execution

• POSIX Environment Model

(43)

Calls into the Environment

if (fork() == 0) { ...

if ((res = recv(sock, buff, size, 0)) > 0) { pthread_mutex_lock(&mutex);

memcpy(gBuff, buff, res);

pthread_mutex_unlock(&mutex); }

...

} else { ...

pid_t pid = wait(&stat); ...

(44)

fork()

Program Under Test

Environment

(C Library / OS)

Environment Model

(45)

fork()

Program Under Test

Environment

(C Library / OS)

Environment Model

Model Code

Symbolic Execution Engine

Equivalent functionality

Executable symbolically

(46)

Starting Point

Symbolic Execution Engine

Network

Stubs

Files

POSIX

Single

-thr

eaded

isola

ted nodes

Single

-thr

eaded

utilities

(47)

POSIX Environment Model

Symbolic Execution Engine

Network

TCP/UDP/UNIX

Files

Threads

Pipes

pthread_*

Processes

POSIX

M

essage passing

Ser

vers and clien

ts

M

ulti-thr

eaded

pr

og

rams

Distr

ibut

ed sy

st

ems

Signals

Asynchr

onous ev

en

ts

,

IPC

Single

-thr

eaded

utilities

(48)

Key Changes in Symbolic Execution

Multithreading and Scheduling

• Deterministic or symbolic scheduling

• Non-preemptive execution model

Address Space Isolation

• Copy on Write (CoW) between processes

(49)

Symbolic Engine System Calls

• Symbolic engine support

needed for threads/processes

1. Thread/process lifecycle

2. Synchronization

3. Shared memory

Symbolic Engine

System Calls

thread_create thread_terminate process_fork process_terminate get_context thread_preempt thread_sleep thread_notify get_wait_list make_shared

1

2

3

(50)

Outline

• Scalable Parallel Symbolic Execution

• POSIX Environment Model

(51)

Testing Real-World Software

Memcached

GNU Coreutils

Apache

(52)

Time to Reach Target Coverage

printf

Faster time-to-cover, higher coverage values

60% coverage 70% coverage 80% coverage 90% coverage 0 10 20 30 40 50 60 1 4 8 24 48

Time to achieve target coverage [minutes]

(53)

Increase in Code Coverage

0 10 20 30 40 50 0 10 20 30 40 50 60 70 80 90

Additional code covered [ % of program LOC ]

Index of tested Coreutil (sorted by additional coverage)

Coreutils

suite (12 workers, 10 min.)

(54)

Exhaustive Exploration

0 1 2 3 4 5 6 2 4 6 12 24 48 Time to complete exhaustive test [hours] Number of workers

Scalability of exhaustive path exploration

(55)

Instruction Throughput

0.0e+00

2.0e+09

4.0e+09

6.0e+09

8.0e+09

1.0e+10

1.2e+10

1.4e+10

1.6e+10

1.8e+10

1 4 6

12

24

48 Useful work done [ #

of

instructions

]

Number of workers

4 minutes

6 minutes

8 minutes

10 minutes

memcached

(56)

Execute the “whole world” symbolically

Symbolic State

Experimental Setup

Client

Process

memcached/

Apache/

lighttpd

TCP Stream Symbolic cmd. Srv. response

(57)

Symbolic Test Cases

• Easy-to-use API for developers to write

symbolic test cases

• Basic symbolic memory support

• POSIX extensions for environment control

• Network conditions, fault injection, symbolic

(58)

Symbolic Test Cases

Testing HTTP header extension

make_symbolic(hdrData);

// Append symbolic header to request

strcat(req, “X-NewExtension: “); strcat(req, hdrData);

// Enable fault injection on socket

ioctl(ssock, SIO_FAULT_INJ, RD | WR);

// Symbolic stream fragmentation

(59)