Crawling and Detecting Community
Structure in Online Social Networks
using Local Information
Norbert Blenn, Christian Doerr, Bas Van Kester and Piet Van Mieghem
TU Delft - Network Architectures and Services (NAS)
Outline
“In order to find communities in a graph one needs the full graph.”
“Crawling large Datasets like Online Social Networks takes very long.”
Facebook: 901 million (active April 2012), Twitter: Over 140 million (active March 2012) Ideal Crawling with one PC: 1s per request: Facebook 29years, Twitter: 4,5years
1.
Crawling
•
BFS/DFS/RFS
•
Mutual Friend Crawling (MFC)
• the Reference Score
• Performance
2.
Community Detection
•
The Reference Score
•
Compared to well known methods
3.
Conclusion
Crawling Online Social Networks
via Breadth/Depth first Search
standard Breadth First Search
1 2 3 4 5 6 7 8 9 10 11 1 2 3 4 5 6 7 8 9 11 10
What most people do (Random First Search RFS) standard Depth First Search
1 2 3 4 i i2 … … … n i1
But unfortunately Social Networks are not tree like using a BFS/DFS/RFS leads to a sampling
bias by using any of these methods and the fact one has to “wait” until the full graph is crawled to detect communities.
Crawling Online Social Networks
via Mutual Friend Crawling
Our proposed method “Mutual Friend Crawling” (MFC) overcomes this situation by crawling a Graph from any given seed point, Community wise.
MFC is based on BFS/DFS plus one assumption:
• the degree of neighboring nodes is known
• and keeps a “Reference Score” SR
• This in the search trajectory the next node to be next node to visit is the one having the highest SR
Crawling Online Social Networks
via Mutual Friend Crawling
Example:
Starting with node 2: its neighbors are 0,1,3,4 with degrees
the Reference Scores are: 0:0.2, 1:0.2, 3:0.25, 4:0.2 Lets take 4
Crawling Online Social Networks
via Mutual Friend Crawling - Performance
BFS (blue) DFS(green) MFC(red)
How is the reference Score behaving while MFC is traversing the graph.
• As there MFC stays in communities the reference score is always increasing denoting that the community is tightly connected. As soon as there is a drop in SR a new
community is been found.
• This “drop” is largest if an expressed community structure can be found. Otherwise it will be small
Community Detection in OSNs
Community Detection in
Online Social Networks
via Mutual Friend Crawling
Problem of misclassification
• If starting with a hub (11), the nodes 10 and 21 are classified as being in the same community as node 11 (the first community).
Solution:
• after “finishing” a community
check if the nodes in this community should really be in this community
Conclusion
& Future Work
We proposed an algorithm to crawl online social networks community wise
• in order to minimize sampling bias in communities.
• to be able to analyze data while still crawling the network The algorithm detects communities,
• (even for directed and weighted graphs) Future work:
• overlapping communities
• formalism to understand the drop in the reference score in order to catch how structured a graph is. (compared to modularity)
Thank you for your attention
Questions
Delft University of Technology Faculty of Electr. Engineering Dept. of Telecommunication Mekelweg 4 2628 CD Delft The Netherlands Room: EWI 19.240
Crawling Online Social Networks
In order to measure the performance we were looking for “ground truth” datasets
As it is very hard to find some real world datasets where the community partition is known we came up with a “Cluster Graph Generator”
via Mutual Friend Crawling - Performance
1. node generation and slot assignment 2. assigning nodes to clusters
3. creating the links
