Querying and managing evolving code dependency graphs

(1)

David Meibusch & Nathan Hawes

Oracle Labs Australia

June 2016

Frappé: Querying and managing evolving code dependency graphs

Querying and managing

(2)

Safe Harbour

The following is intended to provide some insight into a line of research in Oracle

Labs. It is intended for information purposes only, and may not be incorporated

into any contract.

It is not a commitment to deliver any material, code, or functionality, and should

not be relied upon in making purchasing decisions. Oracle reserves the right to

alter its development plans and practices at any time, and the development,

release, and timing of any features or functionality described in connection with

any Oracle product or service remains at the sole discretion of Oracle.

Any views expressed in this presentation are my own and do not necessarily reflect

the views of Oracle.

(3)

The Truth is in the Source!

But the source is often complicated, multi-language and really really big

(4)

Frappé: code as a property graph

• _{Graph natural in this domain}

–

_{Call graph, directory hierarchy, type hierarchy, data/control flow graphs}

• _{Overlay data from different spaces}

–

_{File system, build, preprocessor, AST, cross-language}

• _{Lets users specify queries in terms of this graph}

(5)

How it works

Frappé: Querying and managing evolving code dependency graphs 5

EXTRACTOR

IMPORTER

EDITOR PLUGINS

CLI SCRIPTS

Source Code

.fo

Server

User

(6)

Graph model example

module

prog

main.c

file

foo.o

module

foo.c

source_file

foo.h

source_file

bar

function_decl

bar

function

main

function

argc

parameter

argv

parameter

input

compiled_from

includes

linked_from

includes

file_contains

calls

has_param

is_type

compiled_from

file_contains

has_param

is_type

declares

int

bar(

int

);

foo.h

#include “foo.h”

int

bar(

int

*input) {

return

input 2;

}

foo.c

#include “foo.h”

int

main(

int

argc,

char

**argv) {

return

bar(&argc);

}

main.c

(7)

How it works

EXTRACTOR

IMPORTER

EDITOR PLUGINS

CLI SCRIPTS

Source Code

IR

Server

User

(8)

Use cases

• _Code search

• _{Code navigation}

–

_{Go to definition}

–

_{Find references}

• _{Code comprehension}

–

_{Visualization}

–

_{Transitive closure calls, includes, etc.}

–

_{Shortest path queries}

(9)

How it looks

• _dddd

• _Web UI Xref

• _{Code Map with impact}

overlay

(10)

Use cases

• _Code search

• _{Code navigation}

–

_{Go to definition}

–

_{Find references}

• _{Code comprehension}

–

_{Visualization}

–

_{Transitive closure calls, includes, etc.}

–

_{Reachability queries}

Queries and Neo4j performance

detailed in GRADES’15 paper:

”Frappé: Querying the Linux Kernel

dependency graph”

(11)

Code search

(12)

Code search

foo.c

(13)

Code search

foo.c

(n with name=‘foo.c’)

struct foo {

int

c;

}

(14)

Code search

foo.c

(c with name=‘foo’)

-[:contains]->

(n with name=‘c’)

UNION

(n with name=‘foo.c’)

struct foo {

int

c;

}

(15)

Code search

foo.c

(c with name=‘foo’)

-[:contains]->

(n with name=‘c’)

UNION

(n with name=‘foo.c’)

struct bar {

int

c;

}

(16)

Code search

foo.c

(c with name ‘foo’)

-[:aliases*]->()-[:contains]->

(n with name=‘c’)

UNION

(n with name=‘foo.c’)

struct bar {

int

c;

}

(17)

Use cases

• _Code search

• _{Code navigation}

–

_{Go to definition}

–

_{Find references}

• _{Code comprehension}

–

_{Visualization}

–

_{Transitive closure calls, includes, etc.}

–

_{Reachability queries}

Queries and Neo4j performance

detailed in GRADES’15 paper:

”Frappé: Querying the Linux Kernel

dependency graph”

(18)

Use cases

• _Code search

• _{Code navigation}

–

_{Go to definition}

–

_{Find references}

• _{Code comprehension}

–

_{Visualization}

–

_{Transitive closure calls, includes, etc.}

–

_{Reachability queries}

• _{Custom user queries}

–

_{(Anti) pattern detection}

–

_{Expose query language}

Queries and Neo4j performance

detailed in GRADES’15 paper:

”Frappé: Querying the Linux Kernel

dependency graph”

(19)

Dependency cycle

source_file

file_contains

source_file

file_contains

source_file

file_contains

references

(20)

Dependency cycle

source_file

file_contains

source_file

file_contains

source_file

file_contains

references

(21)

Dependency cycle

references

()-[

:includes | uses_namespace | expands | interrogates | undefines | aliases |

uses_enumerator | has_friend | isa_type | extends | uses_type | throws |

has_ret_type | has_param_type | calls | may_call | overridden_by | declares

| reads | writes | dereferences | address_of | type_of | size_of | align_of

| casts_to

]->()

(22)

BUT

Developers working off of different

versions of the code

(23)

Target Scenario

• 1000s of developers working from

main branch

• _{Changes merged regularly}

• _{Most developers working off of}

versions from the past 30 days

(24)

Current Deployment

• _{Separate Neo4j instance for the most}

recent

5 versions of the code

• _{New graph generated in nightly}

regression

• _{Script on the client to determine which}

server to connect to

• _{Deployment effort and complexity}

• Inefficient use of resources

–

Redundant data

–

Memory requirements

(25)

Current Deployment

• _{Separate Neo4j instance for the most}

recent

5 versions of the code

• _{New graph generated in nightly}

regression

• _{Script on the client to determine which}

server to connect to

• _{Deployment effort and complexity}

• Inefficient use of resources

–

Redundant data

–

Memory requirements

Single logical server

with multiple versions

and efficient storage

(26)

Use cases for multiple versions

• _{Single version use cases per version}

• _{Code review: 2 versions}

–

_{Are there any architectural constraints being newly violated?}

–

_{Are there any new usages of deprecated methods?}

–

_{Are any methods now unused?}

Select version

(27)

Version selection

SELECT

path

FROM

freebsd@1013

WHERE

path =

(:function WITH name='source') -/:calls*/-> (:function WITH name='sink')

PATH

vcalls := () -[:call WITH 1013 between fromV and toV]-> ()

SELECT

path

FROM

freebsd

WHERE

path =

(:function WITH name='source’, 1013 between fromV and toV)

-/:vcalls*/->

(:function WITH name='sink’, 1013 between fromV and toV)

(28)

Use cases for multiple versions

• _{Single version use cases per version}

• _{Code review: 2 versions}

–

_{Are there any architectural constraints being newly violated?}

–

_{Are there any new usages of deprecated methods?}

–

_{Are any methods now unused?}

Match in each version

+

Compare results

(29)

Compare results

SELECT

DIFFERENCE path

FROM

freebsd@1013, freebsd@1014

WHERE

path =

(:function WITH name='source') -/:calls*/-> (:function WITH name='sink')

SELECT

path

FROM

freebsd@1013

WHERE

path =

(:function WITH name='source') -/:calls*/-> (:function WITH name='sink')

DIFFERENCE

SELECT

path

FROM

freebsd@1014

WHERE

path =

(:function WITH name='source') -/:calls*/-> (:function WITH name='sink')

(30)

Open Questions

• _{Node/Edge identity}

–

_{Supplied or derived using graph information}

• _{Query language expressiveness}

–

_{Regular path expressions}

–

_{Multiple edge labels}

• _{Efficient storage and querying for multiple graph versions:}

–

_{List or ideally DAG of versions}

–

_{Union, intersection, difference of results from different versions}

–

_{Query language that abstracts away versioning}

(31)

FreeBSD dataset available on OTN

• _{Includes graphs of 10.1, 10.2, 10.3}

kernel + documentation

• _Each graph

–

_{Extracted from 10M LOC}

–

_{2 million vertices}

–

_{10 million edges}

• _{Try it out and get in touch:}

–

_{[email protected]}

david [email protected]

[email protected]

http://www.oracle.com/technetwork/oracle-labs/datasets/downloads/index.html

(32)

(33)

Graph model example

main.c

file

bar

function

main

function

name_start_line

name_start_column

use_start_line

use_end_line

name_start_line

name_start_column

use_start_line

use_end_line

#include “foo.h”

int

bar(

int

*input) {

return

input 2;

}

foo.c

#include “foo.h”

int

main(

int

argc,

char

**argv) {

return

bar(&argc);

}

main.c

fil

e_

con

ta

in

s

ca

lls

name location

use location

name location

use location

(34)

Graph model example

_main.c

file

main

function

name_start_line

name_start_column

use_start_line

use_end_line

name_start_line

name_start_column

use_start_line

use_end_line

#include “foo.h”

int

main(

int

argc,

char

**argv) {

return

BAR(&argc);

}

main.c

fil

e_

con

ta

in

s

ca

lls

#define

BAR(A) bar(A)

int

bar(

int

);

foo.h

#include “foo.h”

int

bar(

int

*input) {

return

input 2;

}

(35)

Graph model example

main.c

file

bar

function

main

function

name_file_id

name_start_line

name_start_column

use_file_id

use_start_line

use_end_line

name_file_id

name_start_line

name_start_column

use_file_id

use_start_line

use_end_line

#include “foo.h”

int

main(

int

argc,

char

**argv) {

return

BAR(&argc);

}

main.c

fil

e_

con

ta

in

s

ca

lls

#define

BAR(A) bar(A)

int

bar(

int

);

foo.h

#include “foo.h”

int

bar(

int

*input) {

return

input 2;

}

foo.c

name location