Media Cloud Building Practicalities

Media Cloud

Building Practicalities

MEDIA-TECH Hamburg May 2012

Damien Bommart Director - Product Marketing www.fpdigital.com

Agenda

• Front Porch Digital DIVASolutions Family

• What is a Media Content Storage Management (CSM) system? • Media Storage Technology Choice (Memory/Flash, Hard Disk,

Optical, Tape)?

• Media Storage Format Choice?

• What about Cloud storage for Media? • Media Cloud Services Evolution?

Our DIVAsolutions Family

• Legacy film and video tape asset migration/ preservation

• Fastest, Cost efficient • Full SD and HD support • Preservation and

mezzanine formats generated in parallel

• CSM is the heart of the file based facility

• DIVArchive is the first CSM solution with full AXF support

• Interface to ALL media devices and storage technologies

• 100s of Petabytes currently under DIVArchive management

• Global facility connectivity and distribution via DIVAnet

10

• Direct format, multi-platform content publishing

• Leverage all CSM stored content to distribute and market content

• Simple end user access to all file based assets

• Most efficient and scalable enterprise class workflow SAMMAsolo

SAMMArobot

Analog to Digital

• DIVAsolutions family as a hosted Service in the Cloud

• Service currently in beta

• IT-centric Disaster Recovery, CSM and MAM as a Service

• Provision services on the fly

• Leverages AXF for true asset protection

• High SLA, limitless scalability and globally distributed

• Big Data centric service for a

fraction of the typical costs of S3, Azure, etc. Online Publishing DIVApublish DIVArchive DIVAnet DIVAdirector Content Storage Management DIVApublish

LYNX

Content Storage Management (CSM)

• CSM is an object aware, intelligent storage management solution • CSM is the middleware which resides between any devices which

produce or consume file-based content and commodity storage • CSM is VERY different than HSM solutions (Oracle SAM-FS,

Quantum StorNext Storage Manager, IBM Tivoli Storage Manager, etc.) which do not handle content as “objects”

• CSM solutions in media applications offer “content aware” features such as transcoding, timecode partial restore, asset/object analytics, etc.

• CSM offer direct integration with business systems to fundamentally link processes, resources and metadata with the content repository

CSM

Content Storage Management

Xcode Services Partial Restore Data MoversData Movers_Data Movers Nearline Storage Fast Access (SAN, NAS, DAS)

Nearline Storage Slower Access (SAN, NAS, DAS)

Archive Storage Limitless Scalability

(Data Tape, Cloud)

Offline Storage (Offsite Media) Live Feed capture Ingest Servers Playout Servers IT Servers Editing Systems CIFS, NFS, FTP, etc. Source/Destination Devices MAM Systems PAM Systems News Systems Web Portal Traffic Systems CIFS, NFS, FTP, etc. Human/Business Systems

Mobile OTT OVP

Systems Web VOD Systems CIFS, NFS, FTP, etc. Source/Destination Platforms Web Services Device Interface Lifecycle Mgmt Analytics Services Fixity Services Migration Services AXF Interface Storage Interface Asset Database

Content Storage Management

ONLINE ONLINE

NEARLINE

NEARLINE

ARCHIVE

ARCHIVE

OFFLINE

OFFLINE

VALUE (STORAGE CAPACITY VS COST)

IM M E D IA C Y O F A C C E S S

CSM

CSM

Storage Pyramid

Storage Format Choices

Hard Disk Storage Flash Storage Optical Storage Data Tape Storage

Pros •Fast transfer speeds

•Good storage density

•Random access media

•Fast mechanical times

•Multiple R/W streams

•Random access

•Persistent storage

•Low energy costs

•Acquisition format •Fast access •Rugged • Random access • No contact read/write •Acquisition format • Emerging holographic

• Low replication costs

• Fast transfer speeds

• Storage density

• Rugged media

• Cost per TB

• Extremely portable

• Expansion costs

• Low replication costs

Cons •Tied to host chassis

•Not portable

•Cost per TB

•Cost of ownership

•Highly mechanical

•Expansion costs

•High replication costs

•Limited read/write

•Cost per TB

•Proprietary formats

•Low transfer speeds

•Low storage density

• Low storage density

• Low transfer speeds

• Questionable shelf life

• Portability

• Single R/W stream

• Sequential data access

• Head and tape wear

• Slower access

How Are My Valuable Assets Stored?

• We must take steps to ensure long term accessibility to valuable file-based assets stored in the CSM

• Key goals of the “ideal storage format” include:

o Ensure long term accessibility

o Self descriptive assets and self descriptive storage media

o Preservation (Open Archival Information System - OAIS) features such as fixity, provenance, etc.

o File encapsulation to wrap related metadata and files

o Scalability for any number of elements of any size and type o Standardized regardless of storage media technology

o Transportability and compatibility between systems

What About TAR?

• Tape ARchive (TAR) format has been around for many decades • Despite following established standards there is no true universal

TAR implementation

• TAR is a legacy format which cripples most of the core values of a Content Storage Management system (e.g. No Partial Restore) • TAR does not maintain on-media catalog for content stored

• TAR does not include resiliency feature

• TAR does not include long term preservation features as defined in the Open Archival Information System (OAIS) model

What About LTFS?

• The Linear Tape File System (LTFS) is a simple file system for linear data tape

• LTFS makes data tapes appear as “removable storage” • LTFS is very useful as a physical file-based transport

mechanism but not for long term storage or preservation

• LTFS offers no media encapsulation and relies on simple folder hierarchies to form important asset relationships

• LTFS cannot scale as it has no support for spanning across storage media limiting file collection sizes

• LTFS can only support modern data tape technologies and is not applicable to other storage technologies

Archive eXchange Format (AXF)

• AXF was developed by Front Porch Digital and provided to the community for standardization and wide-scale adoption

• AXF is a fully self-contained, self-describing file storage "container”

• AXF is IT-centric and not tied to media applications only!

• AXF is based on an innovative “file system per object” approach

• AXF fully supports all storage technologies – now and into the future!

Structured Unstructured

Metadata

Proprietary Open

File Payload Ancillary Files

What is AXF?

Components

Provenance Fixity Context Reference Access

Control

Preservation Elements

Universal Storage Agnostic File System

File System

AXF and Storage

AXF is storage technology, operating system and file system agnostic

AXF Object

Spinning Disk (NTFS, FAT, etc.)

Data Tape

(Block Based, LTFS, etc.) Solid State Disk

(NTFS, FAT, etc.)

Flash Media (NTFS, FAT, etc.)

Future Storage Technologies

DVD (UDF, etc.) Blu-Ray (BDFS, UDF, etc.) Holographic (UDF, etc.)

AXF Benefits

• AXF Objects can scale to any size and encapsulate any number of files with full support for media spanning

• No need to upgrade existing storage infrastructures (e.g Tape drives) • AXF guarantees long term compatibility and resiliency with

self-describing features for both AXF Objects and AXF Media

• AXF overcomes all the technical, operational and functional limitations of TAR and LTFS

• AXF is an IT-centric implementation and is not limited to media files alone (documents, imaging data, etc.)

www.OpenAXF.org

• OpenAXF.org is the community portal for the AXF initiative

• The website includes news, documentation, videos and whitepapers providing technical details on AXF

• This is the focal point for the AXF community

o please sign up now!

• AXF standardization work is well underway within SMPTE so please visit www.smpte.org to track its progress

What about the Cloud?

• Amazon, Microsoft and many others offer scalable cloud based storage “services”

• All claim reduced capital costs and high availability via globally distributed data centres

• Unfortunately these solutions are not tuned for “big data” • Amazon Simple Storage Service (S3) cloud storage costs:

o $0.125/GB per month to store data

o No charge for archive transfers (uploads) o $0.10/GB for restore transfers (downloads)

Can Cloud Storage Work?

cannot work in “big data” environments like those typical in the media industry

• However CSM-based solutions “in the cloud” are now available and do make real economic sense

• These solutions offer cost effective “big data” storage as well as the content-centric features of “CSM as a service” • How do different cloud services topologies fit and can the

Customer Facility PROS

• Cost effective content protection • Data tapes can be shipped daily

offering rapid protection

• “FedEx” offers fast transfers! • Can scale to any size with little

additional investment • Capital costs are minimal

• Workflows are very well known…

CONS

• Workflows are human intensive • Recall times can be excessive

• Tapes on the shelf are not “active” and must be managed over time

• Does not offer any redundancy to the main facility

• Main facility disaster can take a

significant amount of time to recover from

Offsite Storage Facility

WAN

Customer Facility A Customer Facility B

PROS

• Private, secure and protected paths • Controlled and fairly predictable

workflows

• Business continuance possible • Shipping of high density data tape

media is FAST and INEXPENSIVE

CONS

• Two or more facilities necessary

• Infrastructure, capital and staffing costs are simple multiples

• Technical challenges are multiplied by the number of facilities

• Heavy capital and operations costs

WAN

Customer Facilities CSM Cloud Service

Provider

PROS

• Protected and encrypted paths

• Controlled and predictable managed workflows

• Business continuance as a service possible

• Shipping of high density data tape media is FAST and INEXPENSIVE • Services can be “tuned” on demand

CONS

• Content must be stored in a “hosted” environment

• Security is a key concern which must be carefully managed

WAN

PROS

• CSM as a service

• Protected and encrypted paths

• Controlled and predictable managed workflows

• Business continuance as a service • Services can be tuned on demand • Global distribution and diversity • Perfect for “elastic” applications

CONS

• Content stored in a “hosted” environment

• Network costs can be prohibitive • High availability WAN must be part

of the solution

• Local “caching” may be necessary to ensure rapid local access

Cloud Service Facilities

CSM Appliance

Disaster Recovery and Global Replication Content Distribution Services

The Connected Enterprise

Business Continuance as a Service Online Video Publishing as a Service Elastic Transcoding Farm

Elastic Facility Services Storage Migration Services

Global Media Asset Management (MAM) Metadata Mining as a Service

Hub and Spoke Distributed Facility CSM in the Cloud

Thank You!

MEDIA-TECH Hamburg May 2012

Damien Bommart Director - Product Marketing www.fpdigital.com