CMU SCS Mining Billion-Node Graphs - Patterns and Algorithms

(1)

CMU SCS

Mining Billion-Node Graphs -

Patterns and Algorithms

Christos Faloutsos

(2)

CMU SCS

Thank you!

•  Panos Chrysanthis •  Ling Liu •  Vladimir Zadorozhny •  Prashant Krishnamurthy

(3)

CMU SCS

Resource

Open source system for mining huge graphs: PEGASUS project (PEta GrAph mining

System)

•  www.cs.cmu.edu/~pegasus

(4)

CMU SCS

Roadmap

•  Introduction – Motivation

•  Problem#1: Patterns in graphs

•  Problem#2: Tools

•  Problem#3: Scalability

(5)

CMU SCS

Graphs - why should we care?

Food Web [Martinez ’91]

>$10B revenue >0.5B users

(6)

CMU SCS

Graphs - why should we care?

•  IR: bi-partite graphs (doc-terms)

•  web: hyper-text graph

• ... and more: D₁ D_N T₁ T_M ... ...

(7)

CMU SCS

Graphs - why should we care?

•  ‘viral’ marketing

•  web-log (‘blog’) news propagation

•  computer network security: email/IP traffic

and anomaly detection •  ....

•  Subject-verb-object -> graph

(8)

CMU SCS

Outline

–  Static graphs

–  Weighted graphs

–  Time evolving graphs

(9)

CMU SCS

Problem #1 - network and graph

mining

•  What does the Internet look like?

•  What does FaceBook look like?

•  What is ‘normal’/‘abnormal’?

(10)

CMU SCS

Problem #1 - network and graph

mining

•  which patterns/laws hold?

–  To spot anomalies (rarities), we have to

(11)

CMU SCS

Problem #1 - network and graph

mining

•  which patterns/laws hold?

–  To spot anomalies (rarities), we have to

(12)

CMU SCS

Graph mining

(13)

CMU SCS

Laws and patterns

•  Are real graphs random?

•  A: NO!!

–  Diameter

–  in- and out- degree distributions

–  other (surprising) patterns

(14)

CMU SCS

Solution# S.1

•  Power law in the degree distribution

[SIGCOMM99]

log(degree)

internet domains

att.com ibm.com

(15)

CMU SCS

Solution# S.1

•  Power law in the degree distribution

[SIGCOMM99] log(degree) -0.82 internet domains att.com ibm.com

(16)

CMU SCS

Solution# S.2: Eigen Exponent

E

E = -0.48

Exponent = slope

Eigenvalue

Rank of decreasing eigenvalue

(17)

CMU SCS

Solution# S.2: Eigen Exponent

E

E = -0.48

Exponent = slope

Eigenvalue

Rank of decreasing eigenvalue

(18)

CMU SCS

But:

(19)

CMU SCS

More power laws:

•  web hit counts [w/ A. Montgomery]

Web Site Traffic Count

(log scale)

Zipf

users

(20)

CMU SCS

epinions.com

•  who-trusts-whom [Richardson + Domingos, KDD 2001] count trusts-2000-people user

(21)

CMU SCS

And numerous more

•  # of sexual contacts

•  Income [Pareto] –’80-20 distribution’

•  Duration of downloads [Bestavros+]

•  Duration of UNIX jobs (‘mice and

elephants’)

•  Size of files of a user

(22)

CMU SCS

Roadmap

•  degree, diameter, eigen,

•  triangles

•  cliques

(23)

CMU SCS

Solution# S.3: Triangle ‘Laws’

(24)

CMU SCS

Solution# S.3: Triangle ‘Laws’

•  Real social networks have a lot of triangles

–  Friends of friends are friends

(25)

CMU SCS

Triangle Law: #S.3

[Tsourakakis ICDM 2008]

ASN HEP-TH

(26)

CMU SCS

Triangle Law: #S.3

ASN HEP-TH

(27)

CMU SCS

Triangle Law: #S.4

SN Reuters

Epinions X-axis: degree

(28)

CMU SCS

Triangle Law: Computations

[Tsourakakis ICDM 2008] But: triangles are expensive to compute

(3-way join; several approx. algos) Q: Can we do that quickly?

(29)

CMU SCS

Triangle Law: Computations

[Tsourakakis ICDM 2008] But: triangles are expensive to compute

(3-way join; several approx. algos) Q: Can we do that quickly?

A: Yes!

#triangles = 1/6 Sum ( _λ_i3₎

(30)

CMU SCS

Triangle Law: Computations

(31)

CMU SCS

Triangle counting for large graphs?

(32)

CMU SCS

Triangle counting for large graphs?

(33)

CMU SCS

Triangle counting for large graphs?

(34)

CMU SCS

Triangle counting for large graphs?

(35)

CMU SCS

EigenSpokes

B. Aditya Prakash, Mukund Seshadri, Ashwin Sridharan, Sridhar Machiraju and Christos Faloutsos: EigenSpokes: Surprising

Patterns and Scalable Community Chipping in Large Graphs, PAKDD 2010,

(36)

CMU SCS

EigenSpokes

•  Eigenvectors of adjacency matrix

!  equivalent to singular vectors

(37)

CMU SCS

EigenSpokes

(symmetric, undirected graph)

N

(38)

CMU SCS

EigenSpokes

N

(39)

CMU SCS

EigenSpokes

N

(40)

CMU SCS

EigenSpokes

N

(41)

CMU SCS

EigenSpokes

•  EE plot:

•  Scatter plot of

scores of u1 vs u2 •  One would expect

–  Many points @ origin –  A few scattered u1 u2 2nd_Principal component

(42)

CMU SCS

EigenSpokes

•  EE plot:

•  Scatter plot of

scores of u1 vs u2 •  One would expect

–  Many points @ origin –  A few scattered u1 u2 90o

(43)

CMU SCS

EigenSpokes - pervasiveness

• Present in mobile social graph

! across time and space

• Patent citation graph

(44)

CMU SCS

EigenSpokes - explanation

Near-cliques, or

near-bipartite-cores, loosely connected

(45)

CMU SCS

EigenSpokes - explanation

Near-cliques, or

(46)

CMU SCS

EigenSpokes - explanation

Near-cliques, or

(47)

CMU SCS

EigenSpokes - explanation

Near-cliques, or near-bipartite-cores, loosely connected So what?

!  Extract nodes with high

scores

(48)

CMU SCS

Bipartite Communities!

magnified bipartite community patents from

same inventor(s) `cut-and-paste’ bibliography!

(49)

CMU SCS

(maybe, botnets?)

Victim IPs?

(50)

CMU SCS

Roadmap

•  degree, diameter, eigen,

•  triangles

•  cliques

(51)

CMU SCS

Observations on weighted

graphs?

•  A: yes - even more ‘laws’!

M. McGlohon, L. Akoglu, and C. Faloutsos Weighted Graphs and Disconnected

(52)

CMU SCS

Observation W.1: Fortification

Q: How do the weights

(53)

CMU SCS

Observation W.1: Fortification

More donors,

more $ ?

(54)

CMU SCS

In-weights ($)

Observation W.1: fortification:

Snapshot Power Law

•  Weight: super-linear on in-degree

•  exponent ‘iw’: 1.01 < iw < 1.26

Orgs-Candidates

e.g. John Kerry, $10M received, from 1K donors

More donors,

even more $

(55)

CMU SCS

Roadmap

(56)

CMU SCS

Problem: Time evolution

•  with Jure Leskovec (CMU ->

Stanford)

•  and Jon Kleinberg (Cornell –

(57)

CMU SCS

T.1 Evolution of the Diameter

•  Prior work on Power Law graphs hints

at slowly growing diameter:

–  diameter ~ O(log N)

–  diameter ~ O(log log N)

(58)

CMU SCS

T.1 Evolution of the Diameter

•  Prior work on Power Law graphs hints

at slowly growing diameter:

–  diameter ~ O(log N)

–  diameter ~ O(log log N)

•  What is happening in real data?

(59)

CMU SCS

T.1 Diameter – “Patents”

•  Patent citation network •  25 years of data •  @1999 –  2.9 M nodes –  16.5 M edges diameter

(60)

CMU SCS

T.2 Temporal Evolution of the

Graphs

•  N(t) … nodes at time t

•  E(t) … edges at time t

•  Suppose that

N(t+1) = 2 * N(t)

•  Q: what is your guess for

(61)

CMU SCS

T.2 Temporal Evolution of the

Graphs

•  N(t) … nodes at time t

•  E(t) … edges at time t

•  Suppose that

N(t+1) = 2 * N(t)

•  Q: what is your guess for

(62)

CMU SCS

T.2 Densification – Patent

Citations

•  Citations among patents granted •  @1999 –  2.9 M nodes –  16.5 M edges •  Each year is a datapoint E(t) 1.66

(63)

CMU SCS

Roadmap

(64)

CMU SCS

T.3 : popularity over time

Post popularity drops-off – exponentially?

lag: days after post # in links

1 2 3

(65)

CMU SCS

T.3 : popularity over time

Post popularity drops-off – exponentially? POWER LAW!

# in links (log)

days after post (log)

(66)

CMU SCS

T.3 : popularity over time

Post popularity drops-off – exponentially? POWER LAW!

# in links

(log) _-1.6

days after post (log)

(67)

CMU SCS

-1.5 slope

J. G. Oliveira & A.-L. Barabási Human Dynamics: The Correspondence Patterns of Darwin and Einstein.

Nature 437, 1251 (2005) . [PDF]

Prob(RT > x) (log)

(68)

CMU SCS

T.4: duration of phonecalls

Surprising Patterns for the Call

Duration Distribution of Mobile

Phone Users

Pedro O. S. Vaz de Melo

,

Leman

Akoglu

,

Christos Faloutsos

,

Antonio

A. F. Loureiro

(69)

CMU SCS

Probably, power law (?)

(70)

CMU SCS

(71)

CMU SCS

‘TLaC: Lazy Contractor’

•  The longer a task (phonecall) has taken,

•  The even longer it will take

Odds ratio=

Casualties(<x): Survivors(>=x)

(72)

CMU SCS

Data Description

"  Data from a private mobile operator of a large city

"  4 months of data

"  3.1 million users

"  more than 1 billion phone records

"  Over 96% of ‘talkative’ users obeyed a TLAC distribution (‘talkative’: >30 calls)

(73)

CMU SCS

(74)

CMU SCS

Roadmap

–  Belief Propagation

–  Tensors

–  Spike analysis

(75)

CMU SCS

E-bay Fraud detection

w/ Polo Chau &

Shashank Pandit, CMU [www’07]

(76)

CMU SCS

(77)

CMU SCS

(78)

CMU SCS

(79)

CMU SCS

E-bay Fraud detection - NetProbe

F A H F 99% A 99% H 49% 49% Compatibility matrix heterophily

(80)

CMU SCS

Popular press

And less desirable attention:

(81)

CMU SCS

Roadmap

–  Belief Propagation

–  Tensors

(82)

CMU SCS

GigaTensor: Scaling Tensor Analysis

Up By 100 Times –

Algorithms and Discoveries

U Kang Christos Faloutsos Evangelos Papalexakis Abhay Harpale

(83)

CMU SCS

Background: Tensor

•  Tensors (=multi-dimensional arrays) are

everywhere

–  Hyperlinks &anchor text [Kolda+,05]

Anchor Text C# 1 1 1 1

(84)

CMU SCS

Background: Tensor

•  Tensors (=multi-dimensional arrays) are

everywhere

–  Sensor stream (time, location, type)

–  Predicates (subject, verb, object) in knowledge base

“Eric Clapton plays

guitar” (48M)

(85)

CMU SCS

Problem Definition

•  How to decompose a billion-scale tensor?

(86)

CMU SCS

Problem Definition

#  Q1: Dominant concepts/topics?

#  Q2: Find synonyms to a given noun phrase?

#  (and how to scale up: |data| > RAM)

(48M)

(87)

CMU SCS

Experiments

•  GigaTensor solves 100x larger problem

Number of nonzero = I / 50 (J) (I) (K) GigaTensor Out of Memory 100x

(88)

CMU SCS

A1: Concept Discovery

(89)

CMU SCS

(90)

CMU SCS

(91)

CMU SCS

Roadmap

–  Belief propagation

–  Tensors

(92)

CMU SCS

• Meme (# of mentions in blogs)

–  short phrases Sourced from U.S. politics in 2008

20 40 60 80 100 120 140 160 0 100 200 Time (hours) # of mentions 100

“you can put lipstick on a pig”

“yes we can”

(93)

CMU SCS 0 50 100 0 50 100 0 50 100 0 50 100 0 50 100 0 50 100 100 100 100

Rise and fall patterns in social media

• Can we find a unifying model, which

includes these patterns?

•  four classes on YouTube [Crane et al. ’08]

(94)

CMU SCS

Rise and fall patterns in social media

•  Answer: YES! 20 40 60 80 100 120 0 50 100 Time Value Original SpikeM 20 40 60 80 100 120 0 50 100 Time Value Original SpikeM 20 40 60 80 100 120 0 50 100 Time Value Original SpikeM 20 40 60 80 100 120 0 50 100 Time Value Original SpikeM 20 40 60 80 100 120 0 50 100 Time Value Original SpikeM 20 40 60 80 100 120 0 50 100 Time Value Original SpikeM

(95)

CMU SCS

Main idea - SpikeM

-  1. Un-informed bloggers (uninformed about rumor)

-  2. External shock at time nb (e.g, breaking news)

-  3. Infection (word-of-mouth)

Time n=0 Time n=nb

Infectiveness of a blog-post at age n:

(96)

CMU SCS

-  1. Un-informed bloggers (uninformed about rumor)

-  2. External shock at time nb (e.g, breaking news)

-  3. Infection (word-of-mouth)

Time n=0 Time n=nb

Infectiveness of a blog-post at age n:

Time n=nb+1

(97)

CMU SCS

SpikeM - with periodicity

•  Full equation of SpikeM

ΔB(n +1) = p(n +1)⋅ U(n)⋅ (ΔB(t)+ S(t))⋅ f (n +1− t)+ε t=n_b n

∑

% & ' ' ( ) * * Periodicity noon Peak _3am Dip

Bloggers change their activity over time

(e.g., daily, weekly,

(98)

CMU SCS

Details

•  Analysis – exponential rise and power-raw fall

Lin-log Rise-part

(99)

CMU SCS

Details

•  Analysis – exponential rise and power-raw fall

Lin-log

Fall-part

(100)

CMU SCS

Tail-part forecasts

•  SpikeM can capture tail part

(101)

CMU SCS

“What-if” forecasting

?

(1) First spike (2) Release date

(3) Two weeks before release

(102)

CMU SCS

“What-if” forecasting

– SpikeM can forecast not only tail-part, but also rise-part!

(1) First spike

(2) Release date

(3) Two weeks before release

(103)

CMU SCS

Roadmap

–  Belief propagation

–  Tensors

(104)

CMU SCS

Scalability

•  Google: > 450,000 processors in clusters of ~2000

processors each [Barroso, Dean, Hölzle, “Web Search for

a Planet: The Google Cluster Architecture” IEEE Micro

2003]

•  Yahoo: 5Pb of data [Fayyad, KDD’07]

•  Problem: machine failures, on a daily basis

•  How to parallelize data mining tasks, then?

(105)

CMU SCS

Centralized Hadoop/

PEGASUS

Degree Distr. old old

Pagerank old old

Diameter/ANF old _HERE

Conn. Comp old _HERE

Triangles done HERE

(106)

CMU SCS

HADI for diameter estimation

•  Radius Plots for Mining Tera-byte Scale

Graphs U Kang, Charalampos Tsourakakis, Ana Paula Appel, Christos Faloutsos, Jure Leskovec, SDM’10

•  Naively: diameter needs O(N**2) space and

up to O(N**3) time – prohibitive (N~1B) •  Our HADI: linear on E (~10B)

(107)

CMU SCS ???? 19+ [Barabasi+] Radius Count ~1999, ~1M nodes

(108)

CMU SCS ???? 19+ [Barabasi+] Radius Count ?? ~1999, ~1M nodes

(109)

CMU SCS ???? 19+? [Barabasi+] Radius Count 14 (dir.) ~7 (undir.)

(110)

CMU SCS

YahooWeb graph (120Gb, 1.4B nodes, 6.6 B edges)

???? 19+? [Barabasi+] Radius Count 14 (dir.) ~7 (undir.)

(111)

CMU SCS

????

Radius Count

(112)

CMU SCS

(113)

(114)

CMU SCS

(115)

CMU SCS

EN

~7

Conjecture:

DE

(116)

CMU SCS

~7

(117)

CMU SCS

(118)

CMU SCS

Centralized Hadoop/

PEGASUS

Degree Distr. old old

Pagerank old old

Diameter/ANF old _HERE

Conn. Comp old _HERE

Triangles HERE

(119)

CMU SCS

Generalized Iterated Matrix

Vector Multiplication (GIMV)

PEGASUS: A Peta-Scale Graph Mining

System - Implementation and Observations.

U Kang, Charalampos E. Tsourakakis, and Christos Faloutsos.

(120)

CMU SCS

Generalized Iterated Matrix

Vector Multiplication (GIMV)

•  PageRank •  proximity (RWR) •  Diameter •  Connected components •  (eigenvectors, Matrix – vector Multiplication (iterated) details

(121)

CMU SCS

Example: GIM-V At Work

•  Connected Components – 4 observations:

(122)

CMU SCS

Example: GIM-V At Work

•  Connected Components

Count

1) 10K x larger

(123)

CMU SCS

Example: GIM-V At Work

Count 2) ~0.7B singleton nodes

(124)

CMU SCS

Example: GIM-V At Work

Count

(125)

CMU SCS

Example: GIM-V At Work

•  Connected Components Count 300-size cmpt X 500. Why? 1100-size cmpt _{X 65.} Why?

(126)

CMU SCS

Example: GIM-V At Work

Count

suspicious

financial-advice sites (not existing now)

(127)

CMU SCS

GIM-V At Work

•  Connected Components over Time

•  LinkedIn: 7.5M nodes and 58M edges

Stable tail slope after the gelling point

(128)

CMU SCS

Roadmap

•  Problem#3: Scalability

(129)

CMU SCS

OVERALL CONCLUSIONS –

low level:

•  Several new patterns (fortification,

triangle-laws, conn. components, etc)

•  New tools:

–  belief propagation, gigaTensor, etc

(130)

CMU SCS

OVERALL CONCLUSIONS –

high level

•  BIG DATA: Large datasets reveal patterns/

(131)

CMU SCS

References

•  Leman Akoglu, Christos Faloutsos: RTG: A Recursive

Realistic Graph Generator Using Random Typing.

ECML/PKDD (1) 2009: 13-28

•  Deepayan Chakrabarti, Christos Faloutsos: Graph

mining: Laws, generators, and algorithms. ACM

(132)

CMU SCS

References

•  Deepayan Chakrabarti, Yang Wang, Chenxi Wang,

Jure Leskovec, Christos Faloutsos: Epidemic

thresholds in real networks. ACM Trans. Inf. Syst.

Secur. 10(4): (2008)

•  Deepayan Chakrabarti, Jure Leskovec, Christos

Faloutsos, Samuel Madden, Carlos Guestrin, Michalis Faloutsos: Information Survival Threshold in Sensor

(133)

CMU SCS

References

•  Christos Faloutsos, Tamara G. Kolda, Jimeng Sun:

Mining large graphs and streams using matrix and

tensor tools. Tutorial, SIGMOD Conference 2007:

(134)

CMU SCS

References

•  T. G. Kolda and J. Sun. Scalable Tensor

Decompositions for Multi-aspect Data Mining. In:

(135)

CMU SCS

References

•  Jure Leskovec, Jon Kleinberg and Christos Faloutsos

Graphs over Time: Densification Laws, Shrinking

Diameters and Possible Explanations, KDD 2005

(Best Research paper award).

•  Jure Leskovec, Deepayan Chakrabarti, Jon M.

Kleinberg, Christos Faloutsos: Realistic,

Mathematically Tractable Graph Generation and

Evolution, Using Kronecker Multiplication. PKDD

(136)

CMU SCS

References

•  Jimeng Sun, Yinglian Xie, Hui Zhang, Christos

Faloutsos. Less is More: Compact Matrix

Decomposition for Large Sparse Graphs, SDM,

Minneapolis, Minnesota, Apr 2007.

•  Jimeng Sun, Spiros Papadimitriou, Philip S. Yu,

and Christos Faloutsos, GraphScope:

Parameter-free Mining of Large Time-evolving Graphs ACM

(137)

CMU SCS

References

•  Jimeng Sun, Dacheng Tao, Christos

Faloutsos: Beyond streams and graphs: dynamic tensor analysis. KDD 2006: 374-383

(138)

CMU SCS

References

•  Hanghang Tong, Christos Faloutsos, and

Jia-Yu Pan, Fast Random Walk with

Restart and Its Applications, ICDM 2006, Hong Kong.

•  Hanghang Tong, Christos Faloutsos, Center-Piece Subgraphs: Problem

(139)

CMU SCS

References

•  Hanghang Tong, Christos Faloutsos, Brian

Gallagher, Tina Eliassi-Rad: Fast best-effort pattern matching in large attributed graphs. KDD 2007: 737-746

(140)

CMU SCS

Project info

(141)

CMU SCS

Cast

Akoglu, Leman Chau, Polo Kang, U Koutra, Danae Beutel, Alex

(142)

CMU SCS

OVERALL CONCLUSIONS –

high level

•  BIG DATA: Large datasets reveal patterns/

CMU SCS Mining Billion-Node Graphs - Patterns and Algorithms

Full text

Mining Billion-Node Graphs -

Patterns and Algorithms

Christos Faloutsos

Thank you!

Resource

Roadmap

Graphs - why should we care?

Graphs - why should we care?

Graphs - why should we care?

Outline

Problem #1 - network and graph

mining

Problem #1 - network and graph

mining

Problem #1 - network and graph

mining

Graph mining

Laws and patterns

Solution# S.1

Solution# S.1

Solution# S.2: Eigen Exponent

E

Solution# S.2: Eigen Exponent

E

But:

More power laws:

epinions.com

And numerous more

Roadmap

Solution# S.3: Triangle ‘Laws’

Solution# S.3: Triangle ‘Laws’

Triangle Law: #S.3

Triangle Law: #S.3

Triangle Law: #S.4

Triangle Law: Computations

Triangle Law: Computations

Triangle Law: Computations

Triangle counting for large graphs?

Triangle counting for large graphs?

Triangle counting for large graphs?

Triangle counting for large graphs?

EigenSpokes

EigenSpokes

EigenSpokes

EigenSpokes

EigenSpokes

EigenSpokes

EigenSpokes

EigenSpokes

EigenSpokes - pervasiveness

•

Present in mobile social graph

•

Patent citation graph

EigenSpokes - explanation

EigenSpokes - explanation

EigenSpokes - explanation

EigenSpokes - explanation

Bipartite Communities!

(maybe, botnets?)

Roadmap

Observations on weighted

graphs?

Observation W.1: Fortification

Q: How do the weights

Observation W.1: Fortification

More donors,

more $ ?

Observation W.1: fortification:

Snapshot Power Law

More donors,

even more $

Roadmap

Problem: Time evolution

T.1 Evolution of the Diameter

T.1 Evolution of the Diameter

T.1 Diameter – “Patents”

T.2 Temporal Evolution of the