People‐centric sensing
Leveraging mobile technologies to infer human activities
Dr. Christos Efstratiou Computer Laboratory, University of Cambridgep y y gPeople‐centric sensing
“People‐centric sensing will help […] by enabling a different way to sense, learn, visualize, and share information about ourselves, friends, communities, the way we live, and the world we live in” A.T. Campbell et al “The Rise of People‐Centric Sensing”Applications
• Individual activity sensing: fitness applications, behavioural suggestions. • Group activity sensing: groups to sense common activities and help p y g g p p achieving group goals. Eg: assess neighbourhood safety, collective recycling efforts. • Community sensing: large scale sensing, where large number of people have the same application installed. E.g., tracking speed of disease across a city, congestion in city. Nicholas D. Lane, Emiliano Miluzzo, Hong Lu, Daniel Peebles, Tanzeem Choudhury, Andrew T. Campbell, A Survey of Mobile Phone Sensing, IEEE Communications Magazine, September, 2010.History of Sensing Platforms
Building sensors Computer vision
On‐body accelerometers MSP Building sensors Computer vision 1990 2000 2010 instrumenting the environment instrumenting the person instrumenting the mobile phone
People‐centric sensing in the
Computer Laboratory, Cambridge
• Explore the potential of mobile phones as a platform for people‐ centric sensing applications. • Explore the potential of instrumented environments with sensors to detect human activities.Mobile Phone Sensing
• Microphone • Camera GPS • GPS • Accelerometer • Compass • Gyroscope • WiFi • Bluetooth • ProximityProximity • Light • NFC (near field communication)Phone Sensing vs Sensor Networks
Sensor Networks
ll f hPhone Sensing
ll f h • Well suited for sensing the environment • Specialized hardware designed to accurately monitor specific phenomena • All resources dedicated to sensing • Well suited for sensing human activities • General purpose hardware, often not well suited for accurate sensing of the target phenomena • Multi‐tasking OS. Main purposed of the device is to support other li ti • High cost of deployment and maintenance (regular recharging thousands of sensor nodes) applications • Low cost of deployment and maintenance (millions of potential users where each user charges their own phone) But not sure if users will keep you app on their device!Mobile Phone Sensing
• The mobile phone sensing domain is filled with “hacks”, and imaginative techniques that were used to circumvent the limitations of a platform that was designed for a different purpose. • However, manufacturers have started to change direction – In the near future we expect the release of • New hardware platforms that facilitate back‐ground sensing • New OS frameworks that incorporate a general purpose sensing middlewareDevelopment Design Patterns
• Collect data – High sampling rate – Label with ground truth (e.g. user walking Æ data set) • Inference pipeline – Use collected data for training {walking}Sensing Feature extraction Classification
• Mobile Sensing App – Feed back to the user
Resources
• Sensing is resource intensive • The mobile phone’s purpose is to support multiple applications • A mobile phone sensing application needs to maintain a balance between – The amount of resources needed to operateCPU MEMORY STORAGE
BATTERY – The accuracy of the detection that is achieved
Applications
Detecting Emotions
• Inference: – Emotional state, location and co‐location with others • Sensors used: – Microphone, bluetooth, GPS – Map speaking features to emotional state Source: “EmotionSense: A Mobile Phones based Adaptive Platform for Experimental Social Psychology Research” –Ubicomp‘10Adaptive Duty Cycling
Adaptive Duty Cycling
20 40 60 80 100 Accuracy [%] 50 100 150 200 250 Energy (joules) 0 20continuous 50% duty learning 0 50
continuous 50% duty learning
Applications
Detecting Workplace Behaviour
• Inference: – social behaviour and work performance • Sensors: – Integrating phones with sensors in the environmentFusing mobile phones and sensor networks
• Research Question: – Can we improve the performance of mobile phone sensing by linking itwith sensing in the environment? with sensing in the environment? • METIS – Sensing Offloading – Reduce energy consumptions on mobile devices – Opportunistic offloading of sensing to the environment – Support continuous sensing Sensor Network
METIS: Sensing offloading
Mobile Phone Social Sensing Application METISSensing Remote Sensing
Local Sensing
Sensing Task Distribution Social Sensing API
Sensor Mapping Sensor Mapping and Inference Plugins Inference Component
Phone Sensors Network Interface
Sensing Infrastructure
Access Point Infrastructure Communication
Fusing mobile phones and sensor networks
Detecting informal interactions in the work place
• Detecting informal interactions in the work place – Location tracking C ti d t ti – Conversation detection Detected Detecting meetings Conversation Patterns 9 10 11 12 13 14 15 16 17 CalendarTime (Hour of the day)
Detecting collaborations
U.2 U.3 U.9 U.10 U.4 U.2 U.3 U.9 U.10 U.4 U.1 U.8 U.5 U.6 U.7 U.11 U.1 U.8 U.5 U.6 U.7 U.11 Level 2 communities Level 1 communitiesDetecting collaborations
U.2 U.3 U.9 U.10 U.4 U.2 U.3 U.9 U.10 U.4 U.1 U.8 U.5 U.6 U.7 U.11 U.1 U.8 U.5 U.6 U.7 U.11 Level 2 communities Level 1 communities ! "#$ ! "%$ ! "&$ ! "' $ ! "( $ ! ") $ ! "*$ ! "+$ ! "#, $ ! "##$ ! "‐ $ . /012$#$ . /012$&$ 3/04567$#$ 3/04567$&$ 3/04567$' $ Ground truthPeople‐centric sensing in construction
• Applying the same techniques – Detecting individual activities in the workplace Construction Site – Fusing data to understand collaborative activities • Applications – Work practice monitoring and understanding – Health & Safety – Real‐time work scheduling and efficiency • Challenges– Commodity mobile phones not widely usedCommodity mobile phones not widely used
– Specialised sensing technologies for people tracking
