Trends in Video Processing for Mobile Systems
Intel Corporation - Client Computing Group
Jean-Pierre Giacalone – Principal Engineer
Introduction
We are in a phase of acceleration in media technology. This is driven by how we
consume media and use our devices.
We see a race in resolution, color gamut, frame rate, dynamic range.
We observe new domains becoming reality (360 Videos, Augmented Capture,
Virtual Reality)
In this talk we will:
• Understand and acknowledge these evolutions
• Look at key ingredients in Systems and SOCs
• List the associated big challenges and call for actions
Rapid Pace of Innovation in Video Technologies
Driven by a Shift to IP-Based Content Delivery
Sling Box Smart TV DVR 1975 1980 1985 1990 1995 2000 2005 2010 2015 VCR Cable AOL DBS Satellite Yahoo Broadband Google HDTV VOD Apple TV Smart Phone Roku
Amazon Prime Instant Video Chromecast Amazon Fire TV Blue-Ray Tivo 7th Gen Consoles Facebook Twitter YouTube Hulu+ Netflix Amazon Prime HDR Netflix UHD Direct TV 4K Netflix Streaming HBO Go Tablet GoPro Hero HD YouTube 360˚ Video On the Horizon: • Dolby Vision • Netflix HDR • 8K Broadcasts • UHD Blue-Ray
• VR Media
Video Technology race is escalating.
New vectors of visual quality & immersion are emerging.
8K
Coming
Streaming Media Exploding on all Form Factors
• Subscriber to OTT TV services
92.1M 332.2M (‘14-’19)
Juniper Research• By 2019 nearly 80% of internet traffic will
be driven by video
Cisco• Nearly 7 in 8 Americans reported watching
online video, with more than half doing so
daily.
ComscoreVideo usages growing thanks to
several key enablers…
4 3x !
The 4K ramp – Enabling more and better pixels
0% 2% 4% 6% 8% 10% 0 5 10 15 20 2013 2014 2015 2016 2017 2018Mi
lli
on
Uni
ts
Desktop+AIO 4K & 5K Panels1
Volume (Mu) % TAM
• 4K TV and desktop volumes growing rapidly
• OS & Microsoft
*software ready for 4K since Windows
*8.1
• Content creation: Adobe
*, CyberLink
*, DivX
*, etc. ready
• Games/Office/Productivity suites: all ready for 4K
Intel equipping 4K panels All-In-Ones
23.6” Available now
IPS class
4K/60Hz
300nits
1000:1 contrast ratio
1. DisplaySearchWibtek
A24Expect to see 10b SDR first out, then 10b HDR.
Next immediate train we are boarding is 8K.
Why HDR?
Intel Restricted Secret
100 NITS range (per BT709)
400-1000 NITS range
Higher Brightness Dynamic
Better Colors
Ecosystem
HDR provides a significantly better viewing experience.
Start to see aggregation of full eco-system and standards.
Very Bright Spots
Higher details (10b)
Wider Contrast
& Color Range
Codec Convert Compose Display I/F
1 stream, HDR
HDR to SDR
SDR
SDR Monitor
HDR
Codec Convert GFx to HDR HDR Compose HDR Display support Display I/F1 stream, HDR
HDR to HDR
HDR Monitor
HDR
Codec Assemble Convert GFx to HDR HDR Compose HDR Display Support Display I/F2-stream, HDR
HDR
HDR Monitor
HDR (2 images)
SDR GFx
SDR GFx
SDR GFx
Key HDR ingredients and data flows
• Decode: HEVC 10b 420 & possibly 444
• Requires Protected Video Path and
HDCP2.2 encryption to remote display.
• Assemble/Convert
• SDR / HDR conversions
• Decoded Streams combinations
• HDR10/BD-A -> 1 stream
• Dolby Vision -> 2 streams
•
Base layer (BT709)
•
Enhancement layer (BT2020)
Display Engine Media pipe GPU or Display Engine• Blend and HDR Display control
• Scaling in HDR color space and dynamic range (FHD content)
• Blending in HDR color space and dynamic range
8
Welcome to the 360-Video World!
Objective
Letting users capture and interact with videos in a
naturalistic and immersive way
Use Cases
Immersive Video Conferencing, Virtual Reality Content
Creation, Action Cameras, Video Blogging, Remote
Home Monitoring
Typical System Characteristics Ranges
3-8
USB 2.0 Cameras60-150
⁰
HFOV of each camera60-180
⁰
VFOV of each camera30-60%
Overlap in HFOV between two cameras1080p
Capture resolution360-Video ingredients
Intel Core i7-5557U
25-30
FPS
Avg
50
%CPU Load
Avg
60
%GPU Load
The Augmented Reality Promise
Professional
Consumer
Healthcare Remote diagnostics Surgery assistance Design/Eng. Interactive prototyping Data visualizatio n Simulation Ideation assistance and tools Military Combat assistance Remote info, Recon., field HMD Gaming “Game in real life” Shopping Interactive ads and signage Price/ availability alerts Visualization & “try-on” Entertainment Picture-in- “picture” Sports/Fitness Tracking HMD Cycling, motor-cycling, skiing, etc. Aid Language translation Memory assistance Directions Notification s Home repair assistance Industrial Remote diagnostics/ maintenanc e Workflow efficiency (e.g. machine assistance) Education Reading extension Classroom support Enhanced educationMany potential use cases for AR
Leading adopters: Gaming, Entertainment, as purchase
with several ancillary uses following
Contextual Sensing the World
11 Light Acceleration Location Orientation UV ShapesGuidance
Enablers:
• Always-on, ultra low power sensing and I/O (camera)
• Effective workload partitioning based on available processing
resources
• Context primitives & Fusion of information
Fusion
Algorithms
HARD SENSING
SOFT SENSING
Activity
Location
Environment
Device Usage
Calendar
Picture Credits: Recon, Fitbit
Raw Data Feature Extraction Classification Inference
•Std dev (x,y,z) •Mean, min, max
(gravity angle) Decision Tree X Z Y •Running •Sitting •Walking Accelerometer •MFCC •Std dev •Dynamic range •Bandwidth
GMM ••ChattingListening to music •Commuting Audio
Video •Intensity gradients ••John is in front of meWe are in a room •Contours •Color •Kernel Machine •Bayesian Net •Sparse Bundle Adjustment
Vision SW
Sensor SW pppppppppp
Guidance – Indoor example with AOS
12
Indoor Guidance is a very promising domain:
• User- or Drone-level navigation support inside a building
• No GPS available (although WiFi localization is potentially present)
• Approaches like Google* Tango couple sensors and vision together
• Optimal partitioning with low power operations is key
Pictures Credits: Google
Camera 1 Camera 2 Camera 3 Multi-cameras I/F IPU GPU CPU Sensors Hub HAL Processing Primitives OpenVX Camera APIs Application-level APIs
