PSI and Big Data
OGCIO Seminar on PSI/Data.One
Jiannong Cao
Department of Computing
TB (10
12bytes) → PB (10
15bytes) → EB (10
18bytes)→ ZB (10
21bytes) ...
The World creates about 2.5EB data every day
The total amount of global data reached 2.8ZB
—
report by International Data Corporation(IDC) in Dec. 2012
Where does the data come from
Internet & Web
Sensing & IoT (M2M)
Enterprises
Government
Answer formerly unanswerable questions
Traditional applications but with real-time processing of big
volume of data, faster and more accurate
Formulate new questions
New applications where solutions exist only when volume of
data becomes big enough
Discover invisible knowledge
New knowledge is discovered from different separate
sources of data
Make evidence based predictions / decisions
In particular, shift from causality (why) to correlations (what happens)
Difficulties in big data research
Cannot get the data
Lack of effective means for
analyzing the data
Limited use of the data
PSI provides open data sets for big data research
Big Data
Analyze
Acquire
Organize Decide
Hong Kong
Beijing
Shanghai
15 Categories
Air pollution indices
Hong Kong PSI Dataset
Buildings Food and environmental
hygiene
Law and order
Charitable fund-raising
activities Geo-referenced public facility data
Image resources Marine News and
information Population Census statistics
Public transport Property market
14 categories
Beijing PSI Dataset
Tourism Dining Transportation Healthcare Entertainment
Shopping Religion Education Social Welfare Living Services
Shanghai PSI Dataset
7 categories
Enterprise
Healthcare
Transportation
Law and Order
Statistics
Property
Environment
PSI Datasets
PSI
Data Format
Data Access
Development
Support
Beijing
files
(e.g. csv, XML)
Direct download
API for map,
transport, search
Shanghai
files
(e.g. csv, XML)
Direct download
N/A
Hong Kong files
Opportunities for Big Data Research
Tourism
Education
Healthcare
Green City
Easy Life
Business
Strategy
Tourism
Geo-referenced
Public Facility Data
•
Public Toilet
•
Free Wi-Fi
Food and Environmental
Hygiene
•
Quality Dining
Shopping
Malls
•
Shopping
Public
Transport
•
Cheapest Route
Green City
Air Pollution Indices
•
Anti-Pollution
Action
Real-time Traffic Data
•
Traffic Scheduling
Business Strategy
Buildings
•
Investment
Property Market
Statistics
Food and Environmental
Hygiene
•
Retail Store Planning
•
Market Research
Population Census
Statistics
Easy Life
Air Pollution
Indices
•
Route Plan
Weather
Data
•
Parking
•
Leisure
Geo-Referenced Public
Facility Data
Real-time Traffic
Data
PSI-enabled Big Data Research
Air Pollution Indices
Data
Real-time Traffic Data
Big data analytics for
Big data analytics for urban transportation
Urban transportation planing & design
With PSI on real-time traffic data, we can
mine the correlation between two roads
detect the congestion points
evaluate the effectiveness of urban traffic planning
Intelligent transportation system
With PSI on real-time traffic data, we can
analyze congestion state of traffic flows
provide a user with the practically fastest route to a
destination at a given time
Urban air quality estimation
With the PSI on real-time traffic and air pollution data, we can
learn the historical and real-time air quality data
estimate the fine-grained air quality information based on
limited number of monitoring stations
predict air quality information throughout the city in future
Joint analysis on urban air quality & traffic
With the PSI on real-time traffic and air pollution data, we can
find out the relationship between urban air quality & traffic
estimate the fine-grained air quality with traffic information
calculate maximal traffic volume with air quality constraints
Air quality monitoring stations in HK
Air quality statistics of areas in HK
0 20 40 60 80 100 120 Jan Feb Mar Apr May Jun Ju l Aug Sep Oct Nov Dec
Roadside API of Mong Kok in HK
2012 2011 Months 10 20 30 40 50 60 70
General API of Yuen Long in HK
2012 2011 0 20 40 60 80 100 120 Jan Feb Mar Apr May Jun Ju l Aug Sep Oct Nov Dec
Roadside API of Central in HK
2012 2011 Months 10 20 30 40 50 60 70
General API of Sha Tin in HK
2012 2011
Example of joint analysis on air quality & traffic
0 20 40 60 80 100 120 Jan Feb Mar Apr May Jun Ju l Aug Se p O ct N ov Dec
Roadside API of Mong Kok in HK
2012 2011 Months 0 20 40 60 80
General API of Yuen Long in HK
2012 2011 Months 0 20 40 60 80 100 120 Jan Feb Mar Apr May Jun Ju l Aug Se p O ct N ov Dec
Roadside API of Central in HK
2012 2011 Months 10 30 50 70
General API of Sha Tin in HK
2012 2011
