ASPERA HIGH-SPEED TRANSFER
SOFTWARE
PRESENTERS AND AGENDA
PRESENTER
Michelle Munson
Aspera President and CEO
michelle@asperasoft.com
• Aspera Overview
• Aspera Technology
• Use Cases in Life Sciences
• Aspera + Intel
• Product Overview and Demos
AGENDA
WHO IS ASPERA?
Software technology company innovating new data transfer solutions
Based in Emeryville California
Founded in 2004, privately held, now with 145 employees
Creators of the fasp™ protocol
• Innovative, patented, highly efficient bulk data transport technology • Unique and core to Aspera’s high-performance file transfer software suite • Outperforms software and hardware WAN acceleration solutions
• Ranked first in every WAN transfer throughput benchmark
Patents:
fasp Bulk Data and Dynamic Bandwidth Control issued in USA and 30 other countries, others pending in over
32 countries
Growth:
over 2,100 customers, over 18,000 licenses, 50% year-over-year growth
Markets Served:
Media and Entertainment, Federal Government, Life Sciences, Healthcare, Cloud Computing,
Software and Gaming, Financial Services, Legal, eDiscovery, Engineering, Technology, Telecommunications, Service
Providers, Architecture and Design, Enterprise IT
Global 24x7 Support:
Support and sales offices in Sophia-Antipolis, Singapore, Virginia US, and Direct Sales and Sales
Engineering throughout globe
ASPERA’S MISSION
Creating next-generation transport technologies
that move the world’s digital assets at maximum speed,
TRENDS IN TECHNOLOGY
Big Data Explosion
•
90% of data today file-based or unstructured
•
Mix of file sizes—but larger and larger files the norm
Diversity of IP Networks—Media, Bandwidth Rates, and Conditions
•
Variable bandwidth rates (slow to super-fast)
•
Bandwidth rates increasing—costs decreasing
•
Network media remains diverse (terrestrial, satellite, wireless)
•
Conditions vary—all networks prone to degradation over distance
Global Workflows—moving Big Data over WANs
•
Teams are geographically dispersed
•
Over distance, network conditions degrade
•
Contemporary TCP acceleration solutions not designed for big data transfer and replication
Cloud Computing Grows Up
•
Amazon Web Services (AWS) S3 cloud storage – 2010: 262 billion objects, 2012: 1.3 trillion objects
•
More choices: Microsoft Azure, OpenStack, HP Cloud
•
No longer a niche – Netflix (transcoding), MTV (global video distribution), BGI (genomic sequencing)
WHAT CONSTITUTES “BIG DATA”
SAMPLE LIFE SCIENCE CUSTOMERS
Transcoding
Public and Private Research Institutions
BIG DATA CLOUD – WHAT’S THE PROBLEM?
CHALLENGES WITH TCP AND ALTERNATIVE TECHNOLOGIES
Distance degrades conditions on all networks
•
Latency (or Round Trip Times) increase
•
Packet losses increase
•
Fast networks just as prone to degradation
TCP performance degrades with distance
•
Throughput bottleneck becomes more severe with
increased latency and packet loss
TCP does not scale with bandwidth
•
TCP designed for low bandwidth
•
Adding more bandwidth does not improve throughput
Alternative Technologies
•
TCP-based - Network latency and packet loss must be low
•
Modified TCP – Improves TCP performance but insufficient for fast networks
•
UDP traffic blasters - Inefficient and waste bandwidth
•
Data caching - Inappropriate for many large file transfer workflows
•
Data compression - Time consuming and impractical for certain file types
•
CDNs & co-lo build outs - High overhead and expensive to scale
FASP
™ — HIGH-PERFORMANCE DATA TRANSPORT
Maximum line-rate WAN transfer speed
•
Transfer performance scales with bandwidth independent
of transfer distance and resilient to packet loss
•
Optimal end-to-end throughput efficiency
Congestion Avoidance and Policy Control
•
Automatic, full utilization of available bandwidth
•
On-the-fly prioritization and bandwidth allocation
Uncompromising security and reliability
•
Secure, user/endpoint authentication
•
AES-128 cryptography in transit & at-rest
Scalable management, monitoring and control
•
Real-time progress, performance and bandwidth utilization
•
Detailed transfer history, logging, and manifest
Enterprise-Class File Delivery
•
Transfers up to thousands of times faster than FTP/HTTP(S)
•
Precise and predictable transfer times
•
Extreme scalability (concurrency and throughput)
FASP 3
– UNIVERSAL HIGH-SPEED DATA TRANSPORT
CLOUD
HYBRID
ON PREMISE
fasp 3
DISTANCE INDEPENDENT
INFRASTRUCTURE AGNOSTIC
Locate your big data anywhere and move it seamlessly with high-speed, secure transfers
FILE SIZE INDEPENDENT
Maximum transfer speed, optimal bandwidth utilization and maximum I/O
throughput on any storage platform
regardless of number or size of files
ALL PARADIGMS & DEPLOYMENT MODELS
Transfer, replicate or synchronize your data in all types of scenarios: one-to-one, hub and spoke,
FASP
TM– PERFORMANCE BREAKTHROUGH
FTP
Across US
US – EU
US – ASIA
Satellite
1 GB
1 – 2 hrs
2 – 4 hrs
4 – 20 hrs
8 – 20 hrs
10 GB
15 – 20 hrs
20 – 40 hrs
Impractical
Impractical
100 GB
Impractical
Impractical
Impractical
Impractical
fasp™
2 Mbps
10 Mbps
45 Mbps
100 Mbps
200 Mbps
1 Gbps
1 GB
70 min.
14 min.
3.2 min.
1.4 min.
42 sec.
8.4 sec.
10 GB
11.7 hrs
140 min.
32 min.
14 min.
7 min.
1.4 min.
100 GB
23.3 hrs
5.3 hrs
2.3 hrs
1.2 hrs
14 min.
Aspera transfer times shorten linearly with bandwidth
Independent of packet loss, delay (network distance)
Cross US – Add 1% to 5% Intercontinental – Add 1% to 10% Satellite – Add 1% to 10%
TCP transfer times limited by packet loss, delay (network distance) NOT BANDWIDTH
HIGH SPEED TRANSFER TO CLOUD STORAGE
WITH DIRECT-TO-CLOUD
Cloud
CUSTOMER CASE STUDIES
Challenge
• Moving and sharing genomic research data to investigators and research scientists worldwide
Solution
• Aspera Connect Server embedded in “EasyGenomics™” to make NGS analysis easier and faster
Results
• Transferred 24 Gigabytes of genomic data from Beijing to California in under 30 seconds • Eliminated the costs and risks of shipping hard drives around the world
Challenge
• Moving large genomics data from clinical providers to GenoSpace’s cloud-based analytics platform running on Amazon Web Services
Solution
• Aspera On Demand integrated with the GenoSpace portal, moves genomic data to the cloud at high speed, maintaining patient privacy and security with encryption in transit
Results
• Cloud-ready solution saves the cost and implementation effort of infrastructure build-outs.
• Aspera will scale to accommodate the large and ever-growing genomics data that GenoSpace receives
High speed connection Web portal, Simple UI Database Computational resources Algorithms workflows
EasyGenomics™ is a cloud-based software as a service (SaaS) platform for bioinformatics data analyses
Cloud-based solution for data analysis
CUSTOMER CASE STUDY: BGI
Challenge
•
Sharing large volumes of data between internationally
dispersed collectors, data analysers and researchers
•
BGI developed cloud-based bioinformatics platform
“EasyGenomics™ to provide scientists with high-speed
data exchange, point-to-click data analysis workflows
including whole genome resequencing, targeted
resequencing, RNA-Seq, small RNA and Denovo assembly
•
BGI needed a high-speed file transfer solution that would
allow users to rapidly upload sequencing data to the cloud
for processing and then quickly download completed
projects, speeding up the data analysis process for
customers
High speed connection Web portal, Simple UI Database Computational resources Algorithms workflows
EasyGenomics™ is a cloud-based software as a service (SaaS) platform for bioinformatics data analyses
Cloud-based solution for data analysis
CUSTOMER CASE STUDY: BGI
Solution
•
BGI deployed the Aspera Connect Server to expand
BGI’s high-speed sequencing data delivery service
•
Integrated Aspera fasp™ high-speed file transfer
technology with EasyGenomics using the rich APIs
available in the Software Development Kit (SDK) and the
Aspera Developer Network (ADN)
•
Provided a web-based file transfer framework that
served as the foundation for the EasyGenomics website,
delivering maximum throughput data movement over any
distance with end-to-end security and exceptional
bandwidth control, regardless of fluctuating network
conditions
June 2012: BGI Demonstrated Genomic Data Transfer at Nearly 10 Gigabits per
Second Between US and China
ACHIEVING 10GBPS+ TRANSFERS OVER GLOBAL WAN
Joint research project between Aspera and Intel Corporation
• Investigate high-speed (10Gbps and beyond) data transfer solutions
• Leverage Intel® Xeon® processor E5-2600 product systems and Aspera’s fasp transport
Hardware Platform
• Intel® Data Direct I/O Technology (Intel® DDIO) which allows Intel® Ethernet controllers to route I/O traffic directly to the processor cache
• Built-in support for Single-Root I/O Virtualization (SR-IOV) which allows virtual machine platforms to bypass the hypervisor in order to directly access resources on the physical network interface
Results
• 300 percent throughput improvement versus a baseline system that did not contain support for Intel® DDIO and SR-IOV, showing the clear advantages of Intel’s innovative Intel® Xeon® processor E5-2600 product family.
• Similar results across both LAN and WAN transfers confirming that Aspera fasp transfer performance is independent of network latency and robust to packet loss on the network
• Approximately the same throughput for both physical and virtualized computing environments
0 2 4 6 8 10 2 3 4 Th ro u gh p u t (G b p s) Number of Sessions
Aspera Transfer Throughput -- Mul ple Sessions
DDIO (LAN) DDIO+SR-IOV (LAN) DDIO+SR-IOV (50ms-0.1% Packet Loss) DDIO+SR-IOV (100ms-1% Packet Loss) DDIO+SR-IOV (500ms-5% Packet Loss) 0 2 4 6 8 10
Baseline DDIO (LAN) DDIO + SR-IOV (LAN) DDIO + SR-IOV (50ms-0.1% Packet Loss) DDIO + SR-IOV (100ms-1% Packet Loss) DDIO + SR-IOV (500ms-5% Packet Loss) Th ro u gh p u t (G b p s)
Aspera Transfer Throughput -- Single Session
17
Results available:
asperasoft.com
Hardware
• Intel Xeon processor E5-2600 family (dual 2.70GHz CPU, 2x16 cores with Intel® HyperThreading Technology)
• NUMA and Intel DDIO enabled as default • Intel Corporation 82599EB
HIGH-SPEED TRANSFER SOFTWARE
Moving the world’s data at maximum speed
ASPERA PRODUCT PORTFOLIO
High-speed transfers
for web, desktop and mobile
High-speed file transfer servers for on premise, private, public, and hybrid cloud deployments
File sharing, collaboration
and exchange applications
Transfer management,
monitoring and automation
Scalable, multi-directional,
multi-node synchronization
TRANSFER
CLIENTS
WEB
APPLICATIONS
MANAGEMENT &
AUTOMATION
SYNCHRONIZATION
TRANSFER SERVERS
Innovative, patented, highly efficient bulk data transport technology, unique and core to all Aspera products
FASP™ TRANSPORT
FASP
™ SOFTWARE ENVIRONMENT
ASPERA DEVELOPER NETWORK
A complete set of SDKs with guides, reference information, and sample code to assist them with integrating Aspera
technology into their own applications. Aspera fasp™ technology can be used in desktop, network-based, and web
applications in place of FTP, HTTP, or custom TCP-based copy protocols.
ASPERA MOBILE APIs
Android SDK
Aspera Android SDK provides a Java API to transfer files using fasp-AIR™.
iPhone SDK
Aspera iPhone SDK with Objective C API to transfer files using fasp-AIR.
ASPERA APPLICATION APIs
faspex™ Web API
The Aspera faspex Web API provides a set of services that enables users to create and receive digital deliveries via a Web interface, while taking advantage of fasp high-speed transfer technology.
ASPERA TRANSFER APIs
Aspera Web Services
A SOAP based web service API that allows initiation, monitoring and controlling of fasp based file transfers.
Aspera Web
Javascript API exposed by Aspera Connect client. It allows integration of fasp based file transfers into web
applications.
Connect 3.0 – New, Enhanced Javascript API
Introducing the new Connect 2.8 developer preview! Integrate the functionality of Aspera Connect 2.8, a fasp-based file transfer client, into your own web applications, while customizing it to your unique brand.
fasp Manager
A class library that allows intiations, monitoring and controlling of fasp based file transfers.
Aspera Multicast SDK
A Java class library that allows initiation and management of IP multicast based data transmissions using Aspera
fasp-MC™.
Console API
The Aspera Console API provides full programmatic management of transfer sessions including initiation, queuing, management and control through a RESTful API.
Shares API
The Aspera Shares API provides full programmatic control over browsing Shares, transfer authorization, and upload / download
ASPERA ENTERPRISE SERVER
UNIVERSAL FILE TRANSFER SERVER
Key Features and Benefits
• Universal base transfer server
• High-speed fasp™ transport of large data sets
• Enterprise-wide high-volume content ingest and distribution • Comprehensive automation and scheduling
• Advanced user and server management • Complete monitoring and reporting
• High-availability configuration (active/active or active/passive) • Deployable on premise or in the cloud (public, private, hybrid)
Applications and Use Cases
• Enterprise-wide file movement • High-volume content ingest• High-performance content distribution
Licensing and deployment
• Multi-tiered licensing based on bandwidth capacity • Usage-based licensing with Aspera On Demand
• High-availability configuration (active/active or active/passive)
• Comprehensive platform support (Unix, Linux, Windows, Mac, Solaris, Isilon OneFS) • Supports connections with Aspera Client, Point-to-Point and other Enterprise Servers
ASPERA CONNECT SERVER
APPLICATION PLATFORM WITH WEB INTERFACE
Key Features and Benefits
• Built on top of the Enterprise Server • Application platform with Web interface
• Simple web portal exposes root directory for each user • Transfers via Aspera Connect web browser plug-in
• Powers fasp™ transfers for Shares and faspex web applications • Rich APIs enable integration into any custom web application
Applications and Use Cases
• Global browser-based file exchange
• Field and third-party content or data gathering • Integration with custom web applications
Licensing and deployment
• Multi-tiered licensing based on bandwidth capacity • Usage-based licensing with Aspera On Demand • Comprehensive platform support
• High-availability configuration (active/active or active/passive) • Supports connections with Aspera Connect browser plug-in client
• Automatic installation of Aspera Connect browser plug-in on the client side
ASPERA CONNECT BROWSER PLUG-IN
Key Features
• High-performance fasp™ transport
• Supports for all major browsers including Chrome, Internet Explorer, Firefox, Safari on Windows, Mac, and Linux (native 64 and 32 bit) • Upload/download client transparently launched by browser • File, directory, and multi-item transfers with queuing • Automatic retry and resume of partial or failed transfers
• Built-in transfer monitor for visual, on-the-fly transfer rate control and monitoring
• Comprehensive transport and content security, integrated with at-rest encryption, easy one-click decryption of files and directories
• Automatic inline installation from the web browser with no administrative rights required
• Rich Javascript API for fully-integrated branded experience, persistence across host reboots, and in browser control
• HTTP fallback mode for highly-restrictive networks with support for authenticated proxies
• Language localized for Japanese, Chinese, French and Spanish
Application examples
• Upload / download client for Connect Server and Aspera faspex™ Server
• Browser based, self installable plug-in enables 3rd party access to content
MULTI-NODE FILE SHARING
Aspera Shares
ASPERA SHARES
– MULTI-NODE FILE SHARING
Key Features
• Web application provides secure access (at global distances) to a consolidated view of all shared content
• Fully integrated Aspera Connect provides maximum speed, secure upload and download of files of any size, at any distance in a lightweight, simple browser plug-in
• Content can be deployed across enterprise servers, private, public and hybrid clouds
• Powerful security model administered through a single
management point combining authorization, user management, and access control
• Search, filtering, and sorting capabilities makes it easy to find individual files or folders in a very large content store
• NEW: Shares are fully integrated as secure, access controlled receiving and publishing points
• Files and directories can be published as faspex packages from any Aspera “Share” to which the user has access. • faspex packages can be delivered directly to any “Share”
Application examples
• Internal distribution of digital assets within the enterprise • Collaboration enablement for geographically-distributed teams • Project team and third-party content or data gathering
ASPERA SHARES FEATURES AND BENEFITS
Powerful security and access model
• Secure access administered through a single management point combining authorization, user management, and access control
• Add and manage users and groups locally, as well as through Active Directory and LDAP
• Complete control over access, such as which nodes are visible, and which nodes, directories and files are accessible • Granular control over all end-user operations such as
browsing, uploading, downloading, making new directories, renaming or deleting files and directories
Easy to use Web interface
• Users can easily navigate across files and folders to locate and initiate a high-speed file upload or download
• A single view consolidates all shared content making it easy to search and navigate across multiple nodes
• Powerful search, filtering, and sorting capabilities simplify finding files or folders in a very large content store
• Operations can be performed on a single file or folder, or directly on the search results
• Easy sharing via simple email
ASPERA SHARES TECHNOLOGY ADVANTAGES
Extreme Scalability
• Web application is decoupled from the content stores and underlying transfer servers or nodes
• Application can be installed on the same node, or on a separate server supporting multiple nodes with a single view of the shared content across those nodes
• Supports transfers of a singe file, individual directories, or batches of files and directories of any size over any distance
Location independence
• Content and transfer nodes can be geographically dispersed across regions, countries, or continents
• Nodes can be deployed in private data centers, public or private clouds, or any combination, with a single Web interface consolidating all content
Flexible Deployment Options
• On-premise, with traditional perpetual software license model
• In the cloud, with Aspera On Demand, as part of a subscription service with a usage based model
• Both software deployments can share content from transfer nodes that are deployed on-premise and in the cloud (hybrid
Easy User Management and Powerful Access Control
• Users and groups can be in AD, LDAP or local (and directories can be “anywhere”)
• Users can have any number of virtual directory access points with individual access control (upload, download, etc.)
fasp Shares Node Node DMZ fasp Datacenter, Emeryville, CA Client, NY, NY THE SOLUTION
• Shares Web app transparently communicates with Aspera server Nodes and displays content in a single user interface
• User browses authorized content across multiple shares
• Independent high-speed data transfers to/from Datacenter, AWS S3, and Windows Azure BLOB, transparent to user
USE CASE: SHARING AND COLLABORATION
HYBRID ACROSS PUBLIC & PRIVATE CLOUDS
29
faspASPERA SHARES - POWERFUL ACCESS CONTROL
ASPERA SHARES - GROUP MANAGEMENT
SAML FEDERATED IDENTITY MANAGEMENT
ASPERA CONSOLE
ASPERA CONSOLE
INTRODUCTION
Challenges
• Effectively manage multi-node Aspera transfer environment • Provide different system administrators with different access
rights to nodes and transfers
• Automate routine repeating transfers
• Transfer reporting and statistics for billing or auditing
Aspera Console
• Web-based application accessible via any standard browser • Consolidated dashboard and detailed drill-downs to monitor
and control transfers in real time
• Automated one-time or recurring Aspera transfers including multi-point "Smart Transfers”
• Remote configuration of nodes’ properties and users • Custom profiles for each Console user
• Comprehensive, centralized transfer history database customizable reports
• Manage any Aspera o Enterprise Server o Connect Server o Point-to-Point Client
• Monitor and control Aspera Sync transfer sessions
ASPERA CONSOLE
NEW FEATURES AND CAPABILITIES
Aspera Console 2.0
•
New look and feel with sortable columns,
searchable pages and improved navigation.
•
New Transfer Queuing service with on-the-fly
reordering and reprioritization control for
transfers on managed nodes via Smart Transfers
and new Console API.
•
New job-level API for comprehensive transfer
initiation, queuing, management and control
through a RESTful API.
•
Fully integrated Aspera Sync reporting including
synchronization progress status, current session
and cumulative statistics on sync’d files and
bytes.
•
Integrated reporting (session views, email
notification, reports) of on-the-fly md5sums for
delivery confirmation.
•
Integrated AD/LDAP support and event-based
audit logging.
•
Enhanced performance specifically for workflows
with hot folder transfers.
•
Full support for monitoring and managing iTunes
ASPERA CONSOLE
MONITOR WITH THE DASHBOARD
36
Dashboard
• Web-based console application
• Centralized view across all managed nodes
• Secure SSH tunnels for automatically connecting to nodes
• Map view showing location and transfers between managed nodes
• Graphical view of inbound and outbound bandwidth usage
• Tabular view of all transfer activity
Activity
• Active, scheduled, recent, and problem transfer activity
• Real time data for active transfers • Track individual site transfers
• One click access to detailed information about each transfer activity and statistics
ASPERA CONSOLE
TRACK INDIVIDUAL TRANSFERS
37
Activity view
• See each transfer’s status and performance • Sort by name, node or time started/completed • Filter by time completed, time scheduled or by status • Search for transfers by transfer name, file name or contact • Drill into each entry for detailed transfer information
ASPERA CONSOLE
ADVANCED AND CUSTOM REPORTING
Project-specific reporting and transfer history
• Choose from a large library of simple and advanced built-in reports
• Reports can be organized, sorted and filtered by virtually any transfer attribute
• Report templates may be scheduled for automatic runs and email delivery
Flexible scheduling
• Run immediately, or schedule reports to run at a later date
Extraction options
• Distribute reports via email • Export to CSV or XML
Custom reporting fields with custom tags
• Custom categories can be derived from metadata assigned to transfers within Aspera applications
Automatic backup and data purge
• Create scheduled backups of console configuration and history data
• Clean up Console data by status and age from the web UI
ASPERA CONSOLE
ADVANCED USER AND NODE MANAGEMENT
39
Manage policies by groups and users
• Define transfer settings and authorization policies by user, including bandwidth caps and priorities, encryption and security settings
• Manage policies by user group, including Active Directory, local Windows and Unix groups
• Filter Aspera Console capabilities by user, group and transfer path, including transfer activity monitoring, bandwidth control, and email notifications
Advanced node configuration
• Configure all node properties including bandwidth controls, encryption settings, directory creation masks, and email notifications
• Configure manifest files, exclusion rules and partial file extension
ASPERA CONSOLE
EMAIL NOTIFICATIONS
40
Define email templates
• Use default templates• Completely change the look and feel with custom templates
• Plain text or HTML
• Custom fields help create useful, descriptive notifications
Associate with activities
• Trigger email notifications from various events
o Transfer starts
o Successful transfer completions o Error conditions
Associate by nodes or by transfer paths
• Specific sources• Specific destinations
ASPERA CONSOLE
ADVANCED BANDWIDTH MONITORING & MANAGEMENT
41
Real time control over active transfers
• Monitor active transfers
• Adjust bandwidth utilization and transfer priority on
the fly in real time
Server-side settings
•
Bandwidth preferences and limits
•
Override client policies
•
Aggregate per-node or per-user groups using
VLinks
Configuring VLinks
•
Aggregate bandwidth targets
•
Including static and scheduled (time-varying) values
•
Using differential bandwidth priorities within a VLink
ASPERA CONSOLE
TRANSFER CONTROL & AUTOMATION
Ad-hoc transfers
•
Remotely browse nodes
•
Initiate ad-hoc transfer jobs from any source to any
destination
Automate distribution with "Smart Transfers”
• Completely control all aspects of the transfer
o Source, connection, transfer rate, security, file handling and notification
• Support single and multi-point distribution
• Fine-grained, to-the-minute scheduling of Smart
Transfers with a repeating schedule
• Keep Smart Transfers private or share with other Console
users
Transfer Processing
•
Automatic forwarding
•
Custom pre- and post-processing functionality on
managed nodes
•
NEW QUEIING SERVICE
ASPERA CONSOLE
DEPLOYMENT
43
Platform Support
•
On-premise installations available for Windows and
Linux
•
Included with Aspera On Demand packages for AWS
o Base Transfer Server
o Application Platform packages
Supports management of
•
Aspera Enterprise Server
•
Aspera Connect Server
•
Aspera Point-to-Point Client
•
Aspera Sync
o Monitor status and performance statistics of sync
sessions
o Per file/directory synchronization status, conflicts,
errors and pending transfers
aspera sync
multi-directional synchronization
aspera faspex server collaborative file transfer
aspera console
network-wide transf er management
aspera enterprise server universal file transfer server aspera connect server universal file transfer server with web interfac e
aspera point- to- point
HIGH-PERFORMANCE FILE
SYNCHRONIZATION AND
REPLICATION
PROBLEMS OF TRADITIONAL SYNCHRONIZATION
Typical Situation
Large numbers of files (1 Million++) to be ‘Syncd’ over the WAN, with time-critical deadlines
Traditional ‘rsync’ - Finding “Changes” is highly costly
•
Each file is compared over the WAN with costly “chatty” messaging: 10 million files x 100 milliseconds RTT
means comparison time is more than 24 hours!
Traditional ‘rsync’ – Uses TCP to transport new data, very slow over the WAN
•
On global WANs (100 millisecond RTT / 1% packet loss and worse) standard TCP rates are <<10Mbps
•
On difficult global WANs (300 millisecond RTT / 3 % packet loss and worse) standard TCP rates are <<1 Mbps
•
Transferring each gigabyte of data takes hours!
•
Opening concurrent rsync jobs has diminishing improvement and is hard to manage
Traditional ‘rsync’ – Moving or renaming directories results in data copies over the WAN
•
E.g. a new “build” directory with the very same files gets copied over the network again. If the directory is large,
can be many hours.
Traditional ‘rsync’ does not have tracking, visibility, bandwidth management
•
The progress is at best to command line standard output on the host where sync’ing occurs
Proprietary synchronization and replication solutions have one or more of the above problems, and
may only work between identical proprietary systems – e.g. Vendor A and Vendor B synchronization
A FUNDAMENTAL SOLUTION
Aspera Sync Solves these Problems
Finding Changes Requires No Remote Communication
•
Changes are computed against a local snapshot with fastest possible local comparison
Uses Aspera FASP for line speed transfers over the WAN with a single transport stream
Moving or renaming directories is replicated as-is at the target
Files with the same checksums are de-duplicated saving network and storage bandwidth
Sync progress, status, and throughput are globally visible and controllable through Aspera Console
Fully interoperable between operating systems and storage systems
INTRODUCING ASPERA SYNC
HIGH-SPEED SYNCHRONIZATION AND REPLICATION
Cross platform software application designed for high performance
synchronization and replication of large file stores over the WAN
Application and use cases
•
Offsite synchronization and replication for storage migration, disaster
recovery and business continuity.
•
Bi-directional system mirroring for alternate access to digital content.
•
Hub and spoke sync for high-speed content collection or distribution.
Features and Benefits
•
High speed, multi-directional synchronization of remote files and directories
•
Highly scalable – designed for today’s extremely large data sets
•
Highly efficient - designed for long distance WANs
•
Secure – matches security standards set by government, FIPS 140-2 compliant
•
Platform agnostic – runs on industry standard Linux and Windows
•
Storage agnostic – compatible with any standard file or block storage system
•
Familiar rsync-like interface shrinks the learning curve for IT professionals
•
One-to-one, one-to-many, and full mesh synchronization
•
Unidirectional, bi-directional; One time, scheduled, or continuous
Scalable, multi-directional,
multi-node synchronization
ASPERA SYNC FEATURES OVERVIEW
High speed, multi-directional synchronization of remote files and directories, solving the fundamental
problems of rsync
•
Built on fasp™ - overcomes TCP bottlenecks for maximum transfer speed regardless of file size, network or
distance
•
Fastest possible resolution of changes - Compares changes to local state (file snapshot), saving costly WAN
chattiness
•
File system notification where available; Superfast scan to detect changes in scan-mode for large incremental
data sets
•
Quick restart after system down
•
Replicates file moves and file renames on the source as a file move or rename on the target; Huge savings not
re-transferring
•
Robust - waits for the system(s) to become stable (i.e., detects whether or not files are still being modified) before
syncing
Robust, Built-in Security
•
User/endpoint authentication, AES-128 cryptography in transit and at-rest data integrity verification and automatic
resume of partial or failed transfers
Platform Support and deployment
•
Command line binary and admin tool for Linux, Windows, and now
OS X and Solaris
•
Available as an add-on license option to Aspera servers and Aspera Point-to-point
ASPERA SYNC
NEW FEATURES AVAILABLE IN 1.5
Integrated Reporting and Management in Console
• Full integration with Aspera Console 2.0 for global visibility into sync sessions and activity
• Reports synchronization progress, per file status, throughput • Enables pause, resume and on-the-fly bandwidth priority
and speed control for Sync sessions
Advanced Features
• Rich include and exclude filters, full support for soft and hard links, configurable user and group ID
• Preserves access time, user and group IDs
• File-level de-duplication eliminates unnecessary transfers, reduces sync times and reduces storage use
• Secure token authorization and document roots as well as API for embedding Sync.
Expanded platform support
• OS X, Solaris and BSD in addition to Windows and Linux
Introduction of distributed file system watcher
• Fastest possible capture of file system changes on NAS (NFS, CIFS) deployments, clusters of ingest servers and very large numbers of files – clusterable and scalable
Disaster Recovery / Business Continuity
• Replicate mission critical data from a primary site to one or more alternate sites
• Systems remain available after a system outage or site loss - critical data is preserved
SYNC USE CASES
Content Distribution / System Mirroring
• Automatically replicate digital content from one system to another
• Content is always available, offering high availability of the service
Hub and Spoke for Collection or Distribution
• Daily collection or distribution of data across a large network of end points
• Continuous update of online content, software and media
Data Migration for Diverse Storage Environments
• Replicate entire file system, or portions of the file
system
• Preserve file attributes such as permissions, access
times, ownership, etc.
Primary Site Alternate Site Primary Site Alternate Site
Web Application
Synchronization time after adding 31,056 files to 1 million small files (100 KB each) over WAN of 100ms/1%
ASPERA SYNC – PERFORMANCE BENCHMARKS
Performance comparison synchronizing many small files (average size 100 KB) over WAN of 100ms/1%
Small files performance
Number of files
Data set size Sync time Throughput
Async 978,944 93.3 GB 9,968 sec (2.8 hours) 80.4 Mbps rsync 978,944 93.3 GB 814,500 sec (9.4 days) 0.99 Mbps Speed up difference 81x Large file performance Number of files Data set size (GB)
Sync time Throughput
Async 5,194 500.1 GB 4,664 sec (1.3 hours) 921 Mbps rsync 5,194 500.1 GB 4,320,000 sec (50 days) 0.98 Mbps Speed up difference 940x
Performance comparison synchronizing many large files (average size 100 MB) over WAN of 100ms/1%
Change file performanc e Number of existing files Number of files added Total size Sync time Throughpu t Async 978,944 31,056 2.97 GB 947 sec (16 min) 26.9 Mbps rsync 978,944 31,056 2.97 GB 37,076 sec (10.3 hrs) 0.68 Mbps Speed up difference 39x Change file performance Number of existing files Number of files added Total size Sync time Throughput Async 5,194 54 5.49 GB 54 sec 871 Mbps rsync 5,194 54 5.49 GB 54,573 sec (15 hrs) 0.86 Mbps Speed up difference 1000x
Synchronization time after adding new files to set of large files (100 MB) over WAN of 100ms/1%
First Run
Second Run
ASPERA DRIVE AND FASPEX 4.0
Next generation file sharing, sync and exchange platform
PROBLEMS WITH CURRENT FILE SHARING SOLUTIONS
Performance
•
Today’s “Drive,” “File Sync” and “File Sharing” services are impractical for Big Data
•
On global WANs (100 millisecond RTT / 1% packet loss+) standard TCP rates are <<10Mbps
•
On difficult global WANs (300 millisecond RTT / 3 % packet loss+) standard TCP rates are <<1 Mbps
•
Transferring or synchronizing each gigabyte of data takes hours!
Data Size
•
Large data sets (large files or large collections of files) cannot move
•
Transfers and sync sessions are extremely slow and most often do not complete at all
Security
•
Security for hosted solutions is dependent on security of the provider
•
Huge concerns over privacy, contractual obligations
Limited deployment options
•
Typical SaaS offerings don’t support on premise or hybrid infrastructure
Limited sharing paradigm
NEW ASPERA SOLUTION
A new file sharing experience
• Brings together the best of Aspera faspex, Shares, and Sync technologies into a single unified platform for file-based collaboration
• Includes full access control, privacy and security of the Aspera fasp technology
• Single file access and control sharing platform across distributed sharing points which may be on premise or in cloud
• Builds on Aspera Connect, Cargo, Sync and Mobile and unifies Shares and faspex paradigms on the backend to achieve a revolutionary file sharing experience
Based on fasp
• Maximum transfer speed fully utilizing the available bandwidth • File size and number independent and distance agnostic
• Top to bottom secure access control, privacy of data and integrity verification across the entire platform, built into fasp including encryption over the wire and at rest
• Direct high-speed transfer with cloud and traditional storage
Aspera Drive and faspex 4.0
• Brings Remote file browsing, transfer, synchronization, and package sending and receiving to the Desktop, Browser and Mobile platforms
• Natural project based collaboration model on the application backend
• Easy to use applications based on familiar models of desktop and web sharing
• Innovative architecture allows for deploying on premise, in the cloud, or in any hybrid combination you choose
ASPERA DRIVE AND FASPEX 4.0
Aspera Drive
• Browsing of remote Projects and authorized shares fully integrated in to the desktop explorer
• Drag and drop files or folders to initiate high-speed transfers to and from remote Shares
• Background file synchronization between desktop and cloud • Integrated faspex package sending and receiving
Next Generation Aspera faspex
• Web application tier and user interface.
• Combines all of the file sharing and package sending capabilities of faspex and Shares to power the Drive file-sharing ecosystem.
• Deploys in conjunction with Aspera Enterprise Server(s) in the Enterprise or in the cloud
• Combines users and groups with file and directory resources into project-based collaboration spaces
• Comprehensive and fine grained access control combining shares with users and projects
Mobile Aspera Drive and faspex Apps
• Extends the experience to mobile platforms
• Browse Projects and Shares and upload and download from iOS and Android devices using fasp.
ASPERA DRIVE AND FASPEX 4.0
INTEGRATED WITH DESKTOP EXPLORER
