Global Multimedia Collaboration System

Global Multimedia

Collaboration System

Wenjun Wu

Indiana University Bloomington IN 47401

[email protected]

Outline

Service-Oriented Collaboration

n

Current main stream real-time collaboration

technologies

• Videoconferencing: H.323, SIP, Access Grid

• Instant messaging & VoIP: MSN/Aol/Yahoo, Jabber, Skype

Service Oriented Collaboration

n

Collaboration

has

• Mechanism to set up members (people, devices) of a

“collaborative sessions”

• Shared generic tools such as text chat, white boards,

audio-video conferencing

• Shared applications such as Web Pages, PowerPoint,

Visualization, maps, (medical) instruments ….

n

b)

and

c)

are “just shared objects” where objects

could be regarded as Web Services

n

We can port objects to Web Services and build a

general approach for making Web services

collaborative

n

a)

is a “Session Service” which is set up in many

Shared Event in Collaboration

• Audio/Video conferencing shares events specifying in

compressed form audio or video

• Shared display shares events corresponding to change in

pixels of a frame buffer

• Instant Messengers share updates to text message streams • Microsoft events for shared PowerPoint (file replicated

between clients) as in Access Grid

n Using Web services allows one to expose update events of all

kinds as message streams

Session Service in Collaboration

Membership: Participant list

n

Role & floor assignment

• Management policy based on shared objects ( audio/video,

text, whiteboard, game

n

Ad-hoc or formally schedule

n

Session is also a “shared meta object” associated with

shared objects

Multimedia Streaming Service

n

Streaming-In



Stream-Out filter

process media “events” between a stream source

and stream sink; and can be

shared

n

composite media service is a DAG ( directed

acyclic graph)

common case ~ a filter chain

n

QoS(in)



QoS(out) ( bandwidth, delay,

jitter,loss)

VoIP &

H.323 Introduction

n Major audio-video standard but broader

n “Binary” format for both “data” and “control” n Supported by many commercial vendors and used

throughout the world in commercial and educational markets

n Supports small-scale multipoint conferences

n Has conference management functionality and the call

signaling functionality H.225 ~ call set-up

H.245 ~ call control

H.323 Protocols

n

H.323 is a “framework” document that describes

how the various pieces fit together

n

H.225.0 defines the call signaling and

communication between endpoints (Call Signaling)

and the Gatekeeper (RAS)

n

Annex G/H.225.0 defines communication between

Border Elements

n

H.245/H.243 is the conference control protocol

Typical H.323 Stack

TCP/UDP TCP UDP

UDP TCP Audio Codecs G.711 G.723.1 G.729 .. Video Codecs H.261 H.263 H.264 .. V.150 T.120 TCP/UD P

T.38 H.225.0Call

Signaling H.245

H.225.0 RAS

Terminal Control and Management Data

Applications Media Control

Multimedia Applications, User Interface

H.323 Architecture

Gatekeeper (security, QoS, routing etc.) MC MP

MC U

H.323 Terminal 1

H.323

Terminal 2 H.323Terminal N

... .

SIP

n

Initially SIP was designed to solve problems for IP

telephony.

n

SIP basic functions

• user location resolution, • capability negotiation • call management.

equivalent to the service H.225 and point to point part of H.245

n

The major difference from H.323

• SIP was designed in a text format and took request-response

protocol style like HTTP

• SIP doesn’t define the conference control procedures like

rtspd Quick-time Gatekeeper SIPUA SIP H.323 RTSP sipd sipcon f _sipu m sip32 3 SIP-H.323 signaling gateway Conferencing Programmable SIP servers Unified messaging Streaming media Hardware SIP phone

Desktop SIP clients sipgw PSTN MGCP SIP-MGCP gateway SIP-PSTN gateway Regular telephones

A Integrated SIP

Sipconf : SIP based Centralized

conferencing

sipc

http://www.cs.columbia.edu/~kns10/software/sipconf

SIP323

SIP/PSTN

n SIP based conferencing server n SIP/SDP and RTP/RTCP

n Audio mixing

n Play-out delay algorithm

n Web based conference setup

n G.711 A and Mu law, G.721, DVI

ADPCM

n Multiple simultaneous

Summary of H.323/SIP

Conferencing Systems

MCU integrates the service of media processing service and session management

n Call-based

A conference call represents control connections between clients and MCUs.

MCU is just a endpoint connected to VoIP softswitch cloud

n Most vendors offer hardware solutions

n Thought as services and controllers but specialized protocols and

implementations; NOT Service-Oriented Architectures!

Access Grid

Access Grid : a large scale audio/videoconference based on a multicast network

n provides the group-to-group collaborations among 150

nodes connected to Internet 2 world wide.

n Use improved MBONE audiovisual tools VIC and RAT n Depends upon high-speed network ( each node needs

20Mbps )

n Peer to peer architecture for distribution with centralized

non standard session control (venue server)

n Did not develop many new capabilities but made existing

Access Grid II

n

Supports multiple screens and dominates some

Proprietary IM

n

MSN/Yahoo/AOL

• Ah-hoc small-group collaboration

• Text, audio-video, gaming and others…. • Remote Presence Service

( typical publish/subscribe messaging application )

• Massive “chat servers” running behind to support millions of

users across the world

• Limited size of buddy list and multi-party meeting • Poor/fair quality for audio/video communication

Skype

Skype: p2p IM&VoIP solution

gained a big success.

• improving sound quality ( use new iLBC audio codec ) from

Global Sound

• Uses a variant of IETF Stun to identify NAT and firewall

• using p2p overlay (Kazaa) rather than expensive, centralized

infrastructure.

• provided supplemental features like instant messaging

service.

• Free on-net VoIP service and a fee-based off-net SkypeOut

service that allows calling to PSTN and cellular phones

Why is Skype so successful?

Better voice quality

excellent audio codec, fancy echo cancellation algorithm

Global IP Sound ( iLBC audio codec )

n

Ability to work behind firewalls and NAT

n

Ease of use ( quite simple UI ) based on IM metaphor

P2P style without centralized MCU

any peer that has enough resource can be selected to host

the mixing service

But they are simply not good enough!

Although all of these systems have advantages, they are not sufficient for building more advanced and integrated

collaboration systems:

n SIP :

had a huge development recently, especially in wireless world very limited supported for conference control

n H.323 :

AV collaboration and T.120 are not well integrated.

the AV communication services and T.120 overlay networks don’t have very good scalability.

H.323 and T.120 are designed in a relative complicated OSI model. It is not easy to understand and develop in their APIs

Most H.323 and SIP conferencing products are based on centralized MCU And no way to take full use of private MCU resources

But they are simply not good enough!

•

Access Grid

heavily depends on multicast service and limited number of uni-cast bridge servers in the Internet 2

No way to be deployed in current Internet

•

Skype : Most promising

use its own propriety protocols and can’t interoperate with other legacy VoIP clients such as H.323 and SIP

only support small-scale audio conferencing ( at most 5-party ) and have no video service

Skype-2 is said to be able to support 10-party in dual-core Intel machines

What’s the ideal videoconferencing system I

n A unified, scalable, robust “overlay” network is needed to

support AV and data group communication over heterogeneous networking environments

• go through firewall and NAT

• provide group communication service in whatever unicast and multicast

networks

• offer reliable data delivery in whatever loss network

• to be configured as P2P or distributed server-based overlay to provide

differential services for VIP and regular users

n A service-oriented architecture for hosting media processing

service and session control service

• More scalable than centralized MCU

• Support various style of conferencing ( massive scale of broadcasting as

well as medium size of private social meetings )

• Service providers can be highly distributed and p2p

~ Skype p2p audio mixing

• Scalable service discovery based on p2p search

What’s the ideal videoconferencing system II

n

A core conference control mechanism is required for

establishing and managing the multi-point conference

• Complete conference control service like T.124 (Generic

Conference Control) in T.120 framework

• more flexible facilities to describe application sessions and

entities ( role-based, XML ) for all kinds of collaboration: audio/video, game, whiteboard

• Session border management

Integrate different AV sessions ( H.323 , SIP, Access Grid, RealStreaming … )

XGSP Web Service MCU Architecture

SIP H323 Access_Grid Native_XGSP

Admire

Gateways convert to uniform XGSP Messaging

High Performance (RTP and XML/SOAP and ..

Media Servers Filters Session Server XGSP-based Control NaradaBrokerin g All Messaging

Use Multiple Media servers to scale to many codecs and many versions of audio/video mixing

NB Scales a distributed

We Services

Break up into “Services”

n Monolithic MCU becomes many different “Simple Services”

• Session Control

• Thumbnail “image” grabber • Audio Mixer

• Video Mixer

• Codec Conversion • Helix Real Streaming • PDA Conversion

• H323/SIP Session/Signaling Gateways

n As independent can replicate particular services as needed

• Codec conversion might require 20 services for 20 streams

spread over 5 machines

n 1000 simultaneous users could require:

• 1 session controller, 1 audio mixer, 10 video mixers, 20 codec

converters, 2 PDA converters and 20 NaradaBrokers

n Support with a stream optimized Grid Farm in the sky

• Future billion way “Video over IP” serving 3G Phones and home media

GlobalMMCS and NaradaBrokering

handled by NaradaBrokering

n Control uses SOAP and codecs use RTP transport n Each stream is regarded as a “topic” for NB

n Each RTP packet from this stream is regarded as an “event” for

this topic

n Can use replay and persistency support in NB to support

archiving and late clients

n Can build customized stream management to administer replay,

and who gets what stream in what codec

n NaradaBrokering supports unicast and multicast

n Use firewall penetration and network monitoring services in NB

NaradaBrokering

Stream

NB supports messages and streams

NB role for Grid is Similar to

MPI role for MPP

Incorporating Support for

Audio/Video Delivery into

NaradaBrokering

n

Added support for an unreliable transport protocol,

UDP

n

Implemented a fixed size (fast) topic (8 bytes).

n

Designed a new compact event with minimum headers.

Added support for legacy RTP clients (both unicast

clients and multicast groups)

n

Improved the routing algorithm to handle real-time

XML based General Session Protocol

The XGSP conference control includes three services:

n

Conference management

supports user sign-in, user create/terminate/join/leave/invite-into XGSP conferences

conference calendar service

n

Application session management

provides users with the service for creating/terminating application sessions, managing session related services such as audio/video mixing

n

Floor control

manages the access to shared collaboration resources in different application sessions

Global-MMCS Community Grid

n This includes an open source protocol independent Web Service

“MCU” which will scale to an arbitrary number of users and provides support for thousands of simultaneous users of

collaboration services.

n The function of A/V media server is distributed using

NaradaBrokering architecture.

• Media Servers mix and convert A/V streams

n Open XGSP MCU based on the following open source projects

• openh323 is basis of H323 Gateway

• NIST SIP stack is basis of SIP Gateway

• NaradaBrokering is open source messaging

• Java Media Framework basis of Media Servers

• Helix Community http://www.helixcommunity.org for Real

Media

Audio/Video Meeting Tests

for single broker

Distributed Brokers Tests

• Going through multiple

brokers does not introduce considerable overhead.

• Scalability of the system can

Analysis of the broker network’s performance

n Test results showed that the broker network can scale well for

both single large size meetings and multiple smaller size meetings.

n In large size meetings, the capacity of the broker network is

increased with respect to the capacity of the added brokers.

n In multiple smaller size meetings, the distribution of users

among brokers are important. Inter-broker stream exchange can reduce the scalability. Few users should not be scattered around the broker network.

n In wide area networks, this videoconferencing system provides

many benefits with distributed broker architecture: bandwidth savings, latency savings, and better quality services.

n In summary, thousands of concurrent users can easily be

media services computation overhead

90% while 4, 23fps stream producers are running Real Streaming

Producer

70% while 50 image grabbers are running Image Grabber

94% while 4 video mixers ( 4-way mixing ) are running

Video Mixing

46% while 20 audio mixers ( six active speakers ) are running

Audio Mixing

Improved JMF Performance

34% - 35% 62% - 63%

40% - 41%

8

31% - 32% 58% - 59%

35% - 36%

7

27% - 28% 51% - 52%

32% - 33%

6

23% -24% 46% - 47%

26% - 27%

5

17% - 18% 40% - 41%

23% - 24%

4

15% - 16% 33% - 34 %

17% - 18%

3

9% - 10% 24% - 25 %

13% - 14%

2

6% - 7% 15% - 16 %

8% - 9%

1

Fast JMF Client Old JMF Client

VIC

Fraction of CPU used versus number of received streams

Polycom, Access Grid

and RealVideo views of

MPEG-4 vs. H.261

n

We added

MPEG4 video

to Java Media Framework

n

Higher quality and flexible video sizes including

Coupled Diverse Streams

n GlobalMMCS supports many diverse streams managed by “video

system”

• Different audio and video codecs

• Shared display using video codecs (MPEG4 or H261) • Motion JPEG – stream of images to and from PDA

n NaradaBrokering represents these and other collaborative

streams just a “topics”; collaboration from multiple clients subscribing to a topic

• Text Chat

• Traditional lossless codec based shared display • White boards

• Control streams

n Streams can be linked to provide composite topics

• eSports project linking video streams and real time annotation

of any frame

eSports Snapshot

Master Video

Annotation Whiteboard

Collaborative Video

Annotation Whiteboard

Integration of PDA, Cell phone and Desktop Grid Access

GlobalMMCS Status/Futures I

n 1. New Collaboration tools

• Shared IDL (Visualization), PowerPoint, OpenOffice (Applications need a month or so more)

• SVG game ( stable ) • Whiteboard ( stable ) • e-Sport ( prototype)

• Jabber IM client ( prototype)

• XGSP needs extension to support

n 2. JMF Audio/Video client ( stable)

• performance enhancement finished

• new codec ( MPEG4-DivX finished; try MPEG4-Xvid and H.264) • support different platform ( Linux, Mac – Mac well developed but

need to chase bug(s) )

GlobalMMCS Status/Futures II

3. Replay & Archive (prototype)

• Replay Engine based on NaradaBroker Storage Service • XGSP-RTSP gateway

• Extend RTSP and NaradaBrokering for Instant Replay

n

4. Web Server Portal ( stable)

• Standard calendar service ( iCalendar, vCalendar) • Flexible conference management

• Need to package UI’s as portlets

n

5. Conferencing Media Processing Service ( Stable)

• Support new codec (H.264 )

• Add DCT domain MPEG4-H.261 transcoder

n

6. H.323 Gateway ( Stable)

GlobalMMCS Status/Futures III

n

7. RealStreaming Gateway ( Stable )

•

Import it to Linux

•

Support Mobile clients

n

8. Global-MMCS deployment & test

•

Core performance measurements complete

•

Test under the setting of multiple NaradaBroker and

NAT/Firewall

•

support deployment for AFRL, NASA, DOE portals

test with remote sites

n

9. SmartPhone Clients (prototype)

n

10. Improved

video codec-based shared display

n

11. S

cheduler of dynamic services

sensitive to

streaming

bandwidth

requirement as well as

CPU

use of codec