Multimedia Playback & Streaming

Multimedia Playback & Streaming

Shadab Rashid

Jam 16

What are you interested in?



Making multimedia apps for…

 Consuming Audio/Video



Dealing with content providers, looking for …

 An application/client solution  A compatible platform

 DRM options

 Protocols & formats supported

What we will cover

 Multimedia API capabilities

 Overview and usage

 C/C++, Qt/QML/Cascades

 Formats & Streaming protocols supported

 Digital Rights Management

 Sample applications & some code

Multimedia API

Overview

Core API capabilities

 Audio/Video playback:

 Local or remote resource (streaming)

 For Video: Use windows within user interface  UI Controls (buttons, progress slider bar….)  Playlist control (next track, previous track …)  Metadata query and setting (artist, album info…)

5

 Audio/video routing:

 Audio via headphones, speakers, TV etc  Second screen via HDMI

Core API capabilities

 Audio volume control:

 Through device buttons or API

 Now Playing Area:

 Set artist, album , get next track/previous track commands

 Invoke core media previewer (or any other ones)

 In “previewer” mode or the “app” mode

 Register to be a media previewer yourself

Native SDK Context

7

C API

• mm-renderer

• BlackBerry Platform Services (BPS)

C++/Qt (QML) - Cascades API

• Wrapper classes

• Easy to use with Cascades UI elements

Multimedia API

C++/Qt (QML)

Some core classes

 bb/multimedia/*.hpp

 And other classes for enum types ….

Let’s see some code

• Play - Pause functionality

• Using QML/Cascades

• And using C++/Qt

MediaPlayer setup (QML)

MediaPlayer control (QML)

MediaPlayer setup - C++/Qt

Event Handling – C++/Qt

Concurrent streams

 You can play numerous streams at the same time

 Create multiple instances of

MediaPlayer

15

A sample app: “RockMediaSample”

Video Playback

 Just create a

ForeignWindowControl

 Attach the window to the

MediaPlayer



Scaling, resizing, repositioning…

 Great performance

 Smooth and responsive on interaction

“VideoPlayerSample”

Beta2 version - already public on:

github.com/blackberry/Cascades-Community-Samples

Full Link:

https://github.com/blackberry/Cascades-Community-Samples/tree/master/VideoPlayerSample

Another sample app!

Creating the

ForeignWindowControl

Attaching the windowId

Multimedia API

C API

mm-renderer overview

 Configure – connect & create context

 Attach audio and video output

 Attach media URI

 Play!

Code Snippets

Event handling – C API

 Done through

BlackBerry Platform Services

(BPS)

.

 Playback state, buffer state, position update

 Network status

 Errors

C API –

some relevant header files

 mm/renderer.h

 bps/mmrenderer.h

 bps/mediacontroller.h & bps/mediaplayer.h

 2 controllers on the device: Buttons & Now Playing service

 Connect to Now Playing service to determine what is playing (meta data,

duration, position)

 Request & control playback for the active media player on the system  Setting metadata

C++/Qt with C BPS API

25

Audio routing

 Use Audio Manager API

 Header files at: audio/*.h

 Direct sounds to other devices:

 Headphones  Speakers

 HDMI TV…and some more

26 Attribution:http://www.iconspedia.com/icon/black-tv-18924.html

Apps heavy with audio effects?

 OpenAl

 Comes bundled with our SDK

 Possible choice for game developers

 Has more granular & low-level audio functionalities  Can be used for cross platform solutions

 IDE even has a sample app!

Formats

Streaming protocols

DRM

Formats

29

Video + Audio

Video

Audio

Image

• 3GP, 3GP2 • M4A • M4V • MOV • MP4 • MPEG-4 • MKV • MP2PS, MP2TS • AVI • ASF, WMV, WMA • AVI • F4V • MP3 • MKA • AAC • AWB • AMR • WAV • OGG (new) • BMP, WBMP • JPG • GIF • PNG • TIFF (new) • SGI • TGA

Codecs

• AC-3 (new: Dolby Digital plus) • MP3 • PCM • AMR-WB • AMR-NB • QCELP • FLAC (new)

• Windows Media Audio (9, 10, 10 ,Professional, Lossless)

• Vorbis • FLAC

Recommended Formats

File format Component Codec

Local: MP4

For remote streaming: Any supported

containers/streams

Video H.264

Audio AAC

31

BlackBerry 10

will support all the formats that

PlayBook supports

and

more

Supported Streaming Protocols

 HTTP

 HTTPS

 HTTP Progressive Download

 RTSP

 HLS v3

HTTP Live Streaming

 Supports live data (unbounded multimedia stream)

 Of course, prerecorded content too

 Client intelligently switches streams (bit rates)

 Based on bandwidth availability

 Allows encryption of media over HTTPS

 Custom DRM schemes for content providers

Digital Rights Management

1) HLS with 128 AES

2) 3

_{party DRM providers}

3) Microsoft PlayReady

4) Adobe Flash Access (FAXS) – web only

34

1) HLS with 128 AES

 On our non-rooted platform, it is impossible for one app to gain

access to the protected content buffers of another

 Internally, it is safe; externally, clear text AES keys are being transmitted in M3U

files.

 If your app can protect the AES keys, then you can use this

approach

 More details on the platform’s security with NDA (for studios, content providers etc)

 The security team can be part of the engagement as well

2) 3

_{party DRM providers}

 RIM has active engagements with some 3

_{party DRM}

providers for supporting BlackBerry 10 for product launch

 RIM can link content providers to these companies

 They will also entertain introductions to other similar companies (that

studios might know or prefer)

 A port to a new platform could take 6 - 9 months for these

companies

 Sooner the process starts, the better

3) Microsoft PlayReady



The client solution would be using RIM’s keys on the

device

 Hence, legal partnership would be required

 Limited to Progressive Download protocol

4) Adobe Flash Access

 Quick solution (for web only)

 Re-using your web infrastructure in the app

Other formats/protocols?

Experience

Media Previewer

 Just want to play a file?

 Invoke the Media Previewer!

 Register your app to be a previewer!

41

Quickly, to the Dev Alpha B!

Unbound invocation

 Just use the URI

Bound invocation

 Set the target as well as the URI

43

For setting the “app” mode specifically, use:

Leverage other apps

 Share links, playlists, artist/album with others

 BBM, Twitter, Facebook

 Calendar events, Universal Search

 Notifications + Universal Inbox

Leverage other devices

 Second screen via HDMI

 Other libraries for wireless communication

 DLNA, UPnP

Don’t forget about the UX!

 User Experience

guidelines

 Peek, Swipe, Gestures  Many more….

Let’s look at an actual app

Nobex Radio (beta)

47

I swear this is the last switch

Ending thoughts ..

 Dive into the IDE and sample apps!

 Documentation sites, header files for multimedia API

 Other relevant sessions:

 Native API Camera – JAM 15, 9:30 AM

 Web API Multimedia – JAM 34, 9:30 AM

 Invocation Framework – JAM 20, 3:30 PM

 BBM – JAM 47 & 49, 9:30 AM & 1:00 PM

 PIM API – JAM 18, 10:45 AM

Questions?

48

You can drop by the Ask the

expert sessions:

• Roundtables: 6 - 7 PM

• One-on-One: Wed 2 – 3 PM • Drop In: Thu 12 to 1 PM

“Live free & code hard !”

Shadab Rashid

@surashid

[email protected]