Building a Network Performance Data Collection Framework with NDT and GIS

(1)

Building a Network Performance Data

Collection Framework with NDT and GIS

Building a Network Performance Data

Collection Framework with NDT and GIS

Seth Peery, Sr. GIS Architect

Virginia Tech Geospatial Information Sciences

April 18, 2011

(2)

Relevance of Performance

Measurement

•

Internet benefits (economic development, health,

(

p

,

education, etc.) are derived from

effective use of

productive applications

[1], not mere

presence

of

infrastructure

infrastructure.

•

Performance measurement allows us to approximate

the user’s experience with the network and identify

y

what applications are feasible

[1] Gurstein, M. (2003). "Effective use: a community informatics strategy beyond the digital

[ ] , ( ) y gy y g

divide." First Monday 8(12).

(3)

Performance Measurement Use Cases

Target Group Outcomes/Rationale

The publicp • Compare measured to advertised performancep p

• Model and estimate commodity network performance

CAIs • Identify underserved institutions

• Benchmarking and impact assessment for projects like US-UCAN

R&E connectors • Monitor R&E network health

Pl f t i i i

• Plan future provisioning

(4)

Supply-side vs. Demand-side

R

t ti

f C

ti it

Supply-side vs. Demand-side

R

t ti

f C

ti it

Representations of Connectivity

Supply Side

pp y

Demand Side

• Map the core and infer the edges

• Locations of infrastructure

• Map the edges and infer the core

• Spatial distribution of users

• Locations of infrastructure

• Service footprints

• Advertised speeds

• Spatial distribution of users

• Adoption and Digital Divide

• User-initiated performance t t

• Typical speeds

• Focus of SBDD data collection

• Passive/automated testing

tests

• Before SBDD, this was all we had

g

4

(5)

Advertised vs. Measured Performance

SBDD Max Advertised Download Speed < 200 mbps 200kbps - 768 kbps 768 kbps - 1.5 mbps 1 5 mbps 3 mbps 1.5 mbps - 3 mbps 3 mbps - 6 mbps 6 mbps - 10 mbps 10 mbps - 25 mbps 25 mbps - 50 mbps 50 mbps - 100 mbps 100 mbps - 1 Gbps > 1 Gbps > 1 Gbps

Virginia Tech NDT Test Data Points under 200 kbps 200kbps - 768 kbps 768kbps - 1.5 mbps 1.5mbps - 3 mbps 3mbps - 6 mbps 6mbps - 10 mbps 6mbps - 10 mbps 10mbps - 25 mbps 25mbps - 100 mbps over 100 mbps 0 0.5 1 2 3 4 Miles

(6)

Sources of User-Initiated Network Performance Data

(7)

Virginia Tech eCorridors Broadband Map

• Started in 2006 • Combines NDT with a Google Maps front-end • Nearly 5000 data points to date, and growing http://www.ecorridors.vt.edu/maps/broadbandmap

(8)

Network Performance Data in GIS

Virginia Data Points under 200 kbps 200kbps - 768 kbps 768kbps - 1.5 mbps 1.5mbps - 3 mbps5 bps 3 bps 3mbps - 6 mbps 6mbps - 10 mbps 10mbps - 25 mbps 25mbps - 100 mbps over 100 mbps 8

(9)

Spatial Analysis of Georeferenced

NDT results

P i t b th

l

d

’t t ll

h

•

Points by themselves don’t tell us much

•

Spatial joins to bounded areas can yield

generalized summary information

•

Getting a sufficiently large sample size is

critical

•

Interpolation holds potential, once certain

constraints are properly modeled

(10)

Effect of AccelerateVirginia Campaigns

Sample Size 0 1 - 5 6 - 10 11 - 25 26 - 100 101 - 644 10

(11)

Generalization by Areal Unit

Vi i i D t P i t

Counties Blocks

Virginia Data Points under 200 kbps 200kbps - 768 kbps 768kbps - 1.5 mbps 1.5mbps - 3 mbps 3mbps - 6 mbps 6mbps - 10 mbps 10mbps 25 mbps 10mbps - 25 mbps 25mbps - 100 mbps over 100 mbps Average Download Avg_downlo no data under 200 kbps 200 kbps - 768 kbps 200 kbps - 768 kbps 768kbps - 1.5 mbps 1.5 mbps - 3 mbps 3 mbps - 6 mbps 6 mbps - 10 mbps 10 mbps - 25 mbps 25 mbps - 100 mbps over 100 mbps over 100 mbps

(12)

Areal Averages used in Analysis

g

y

County-level aggregated map data

County business patterns

12

(13)

Interpolation

Kriging Predicted Values 0.002038659 0.002038659 - 0.2 0.2 - 0.768 0.768 - 1.5 1.500000001 - 3 3.000000001 - 6 6.000000001 - 10 10.00000001 - 20 20.00000001 - 92.98016357 92.98016358 - 100

(14)

Testing Anchor Institutions: v1

(15)

Anchor

Institutions v2 :

Anchor

Institutions v2 :

Internet2 K20

CAIDATA

Institutions v2 :

Internet2 K20

CAIDATA

Project

(16)

Research Directions

• Network performance measurements can be aggregated to any desired spatial unit of analysis: county, ZIP code, Census block

• Data can be analyzed over large regions or at the local level

• Objective, user-contributed network performance measurements can be used as inputs to models that quantify many other

can be used as inputs to models that quantify many other outcomes of network connectivity

• Economic Impact

• Education

• Health Care

• Government Performance

• Safety and SecuritySafety and Security

(17)

Combining Performance Measurement

with Specialized Research Studies

eCommerce Survey Respondents

Speed_Download_Mbps 0.000000 - 0.200000 0.200001 - 0.768000 0.768001 - 1.500000 1.500001 - 3.000000 3.000001 - 6.000000 6.000001 - 10.000000 10.000001 - 25.000000 25.000001 - 100.000000 100.000001 - 319.062000 Health IT Adoption 0 12.5 25 50 75 100 Miles E-Commerce usage by Businesses

(18)

Seth’s Research Design for Economic Impact

g

p

(19)

Challenges of Using Measured

Performance Data in Research

•

Removing the measurement infrastructure as

a source of error

•

Controlling for temporal and spatial variability

in individual speed tests

•

Understanding the limits to the

g

generalizability of individual tests

(20)

Platform Considerations

• Attend Performance WG on

Wednesday at 0730 to learn more

• I2 is a leader in the measurement space due to NDT

• Is “mass market” network performanceIs mass market network performance measurement a fundamentally

different use case?

• The e de facto de acto standard should be opensta da d s ou d be ope

• More fundamentally…

• User-initiated vs. passive / automated testing?

• Moving the test platform closer to the user … regardless of platform

(21)

Contact Information

Seth Peery

Senior GIS Architect, Enterprise GIS

2060 Torgersen Hall (0197), Blacksburg, VA 24061

(540) 231-2178

sspeery@vt edu

[email protected]

http://gis.vt.edu