Using Mobile Processors for Cost Effective Live Video Streaming to the Internet

(1)

Zürcher Fachhochschule

Using Mobile Processors for Cost Effective

Live Video Streaming to the Internet

Hans-Joachim Gelke

Tobias Kammacher

Institute of Embedded Systems

(2)

Agenda

1. Typical Application

2. Available Processors

3. Processor Architectures

4. Interfaces

5. Software Implementation

6. Practical Experiences

7. Further Applications

2

(3)

Typical Video Signal Processing Pipeline

3 Color Space Conversion Scaling Picture in Picture h.264/h.265 Encoder Audio/Video Mux Encryption Transport Protocol Packer Forward Error Correction Recorder Video Input Ethernet Output Audio 2nd Video Source

(4)

Typical Video Signal Processing Pipeline

(5)

Typical Video Signal Processing Pipeline

(6)

Typical Video Signal Processing Pipeline

(7)

Typical Video Signal Processing Pipeline

(8)

Typical Video Signal Processing Pipeline

(9)

Typical Video Signal Processing Pipeline

(10)

Typical Video Signal Processing Pipeline

(11)

Typical Video Signal Processing Pipeline

(12)

Typical Video Signal Processing Pipeline

(13)

How is a Mobile Processor Defined?

4

Audio and Video Processing Units Display Camera Graphics Processing Multi Core CPUs DSP Connectivity Positioning Systems

(14)

How is a Mobile Processor Defined?

4

(15)

How is a Mobile Processor Defined?

4

(16)

How is a Mobile Processor Defined?

4

(17)

How is a Mobile Processor Defined?

4

(18)

How is a Mobile Processor Defined?

4

(19)

How is a Mobile Processor Defined?

4

(20)

How is a Mobile Processor Defined?

4

(21)

How is a Mobile Processor Defined?

4

(22)

How is a Mobile Processor Defined?

4

(23)

How is a Mobile Processor Defined?

4

(24)

How is a Mobile Processor Defined?

4

(25)

How is a Mobile Processor Defined?

4

(26)

How is a Mobile Processor Defined?

4

(27)

How is a Mobile Processor Defined?

4

(28)

How is a Mobile Processor Defined?

4

(29)

How is a Mobile Processor Defined?

4

(30)

The Players

5

TK1 and TX1 X7-Z8700

Snapdragon 810

(31)

Zürcher Fachhochschule Exynos Apple A8

The Players

5 TK1 and TX1 X7-Z8700 Snapdragon 810

(32)

Zürcher Fachhochschule Allwinner A31 RK3288 Exynos Apple A8

The Players

5 TK1 and TX1 X7-Z8700 Snapdragon 810

(33)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4

64-bit ARM A57 2MB L2-Cache

CPU

1 CPU

2 CPU

3 CPU

4

64-bit ARM A53 512kB L2-Cache

(34)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4

Operating

Mode

(35)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4

(36)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4

Sleep

Mode

(37)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4

(38)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4 192/ 256 Core GPU

OpenGL, CUDA, DX12, AEP,

OpenCL

(39)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4

(40)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4 H.264/AVC & H.265/HEVC

4096 × 2160 pix/30 Hz

(41)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4

(42)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4 H.264/AVC & H.265/HEVC

4096 × 2160 pix/60 Hz/10 bit

(43)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4

(44)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4 HDMI 2.0

4096×2160pix,

Display Serial

Interface (DSI)

(45)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4

(46)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Scaling etc.

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4

(47)

Video Interface

Video Encoder

GPU

Video Decoder

Video Processor

Hardware Resources

6

CPU

1 CPU

2 CPU

3 CPU

4 CPU

1 CPU

2 CPU

3 CPU

4

(48)

Mobile Processor Peripherals

7 I2S DSI CSI PCIe I2S SDIO

(49)

Processors available on industrial modules

8

Nvidia TK1 module Snapdragon 810 module _{Allwinner A31 module}

(50)

Mobile Processor

GStreamer:

A Multimedia Framework

9

Generic Multimedia Processing Device

(51)

Mobile Processor

GStreamer:

A Multimedia Framework

9

Color Space Conversion Scaling Picture in Picture H.264/H.265 Encoder Audio/Video Mux Encryption Transport Protocol Packer Forward Error Correction Recorder Video Input Ethernet Output Audio 2nd Video Source

(52)

Mobile Processor

GStreamer:

A Multimedia Framework

9

Video Processing

(53)

Mobile Processor

GStreamer:

A Multimedia Framework

9

Network

Video Processing

(54)

Mobile Processor

GStreamer:

A Multimedia Framework

9

Network Network

Video Processing

(55)

Mobile Processor

GStreamer:

A Multimedia Framework

9

Linux Kernel GStreamer

Network Network

Video Processing

(56)

GStreamer Internals

10

(57)

GStreamer Internals

10

Gstreamer Core Framework Sources:

File Camera TCP/UDP …

(58)

GStreamer Internals

10

Gstreamer Core Framework Sources: File Camera TCP/UDP … Sinks: File Display TCP/UDP …

(59)

GStreamer Internals

10

Gstreamer Core Framework Filters:

Converters Mixers …

Containers:

Audio & Video MP4 / MKV … Transport-Protocols: HTTP/RTSP … Sources: File Camera TCP/UDP … Sinks: File Display TCP/UDP …

(60)

GStreamer Internals

10

Gstreamer Core Framework

Encoders: Audio (MP3 / AAC) Video (H.264/H.265) HW-accelerated … Filters: Converters Mixers … Containers:

(61)

GStreamer Internals

10

Decoders: Audio (MP3 / AAC) Video (H.264/H.265) HW-accelerated … Encoders: Audio (MP3 / AAC) Video (H.264/H.265) HW-accelerated … Filters: Converters Mixers … Containers:

(62)

GStreamer Internals

10

Decoders: Audio (MP3 / AAC) Video (H.264/H.265) HW-accelerated … OS Kernel Encoders: Audio (MP3 / AAC) Video (H.264/H.265) HW-accelerated … OpenMAX Filters: Converters Mixers … Containers:

(63)

GStreamer Internals

10

Decoders: Audio (MP3 / AAC) Video (H.264/H.265) HW-accelerated … OS Kernel Encoders: Audio (MP3 / AAC) Video (H.264/H.265) HW-accelerated … OpenMAX Filters: Converters Mixers … Containers:

Audio & Video MP4 / MKV … Transport-Protocols: HTTP/RTSP … Sources: File Camera TCP/UDP … Sinks: File Display TCP/UDP … Access to HW-accelerated CODECs

(64)

OpenMAX (Open Media Acceleration)

11 Modules / Drivers Video DMA HW accelerated Video Encoder HW accelerated Video Decoder

CPU Mobile Processor

Linux Kernel Libraries

Bellagio (OpenMAX IL)

Software

Kernel Space Gstreamer Plugins

OpenMAX (OMX)

GStreamer Core

Multimedia Application Software User Space

(65)

Macro Block vs. Quad Tree Procedure

12

Quad Tree Procedure

Macro Block Procedure

H.264

H.265

(66)

Comparison between H.264 and H.265

Video Quality

13

(67)

Comparison between H.264 and H.265

Video Quality

13 H.265 11Mbps H.264 19Mbps

(68)

Comparison between H.264 and H.265

Video Quality

13 H.265 11Mbps H.264 19Mbps H.265 30Mbps H.264 67Mbps PSNR vs. Bitrate for H.264 and H.265

(69)

Your Mobile Processor Application ?

14 Mobile Processor Android TV Video Conferencing Lecture recording streaming Medical Imaging Driving Assistance Source: Google Source: Google Source: PMK