Data Explosion
At the 2010 “Techonomy” conference, Google CEO Eric Schmidt asserted that we are now generating more
data every two days than we generated between the dawn of civilization and 2003. While this particular assertion received a great deal of (well-‐deserved) attention, his next point was no less significant: users, such as you and I are, the primary contributors to this explosion in data. That’s right—our email, Facebook, Twitter, LinkedIn, instant messaging, files, SMS messages, videos, and pictures are the culprits, and nowhere are the implications felt more keenly than in IT.
We in IT have struggled for years to address the ever-‐growing impact of user-‐generated data. Email is a great example. Over the past ten years, mailboxes have grown from several megabytes to hundreds of megabytes and now the average corporate mailbox is allocated over 10 gigabytes of storage—and the trend shows no sign of relenting. Organizations now want to provide users with “unlimited” mailboxes that could contain hundreds of gigabytes of data.
We are also tasked with managing new data types. Priorities related to improved collaboration, data loss prevention, and intellectual property protection are driving us to move files from client storage (laptop and desktop hard drives) to IT-‐managed storage and from simple network folders to complex systems such as Microsoft SharePoint and EMC Documentum. The same issues are driving standardization around IT-‐ managed instant messaging systems (like IBM Sametime and Microsoft OCS) rather than public IM (like MSN or Yahoo Messenger). And IT is now being called upon to address the unmitigated rise of social media, blogging, and mobile communications as business tools.
Regulations and Their Consequences
The explosion in IT-‐managed data has occurred during a period when regulators are placing new requirements on the handling, retention, and disposition of content. For example:
● The United States Federal Rules of Civil Procedure (FRCP) require that organizations of all sizes maintain data archives that are readily accessible in the event of litigation.
● The Sarbanes-‐Oxley Act (SOX) requires that companies preserve a variety of correspondences (including email messages) for a period of seven years.
● The Financial Regulatory Authority (FINRA) and Securities and Exchange Commission (SEC) place numerous restrictions on financial services firms related to the management and preservation of email, instant messaging, and social media data.
● The Health Insurance Portability and Accountability Act (HIPAA) requires that companies operating in the healthcare industry retain certain communications and documentation (which can include email messages and attachments) for a minimum of six years.
The list goes on and on. In addition to the examples cited above, industries ranging from energy to state and federal government to education and the non-‐profit sector all fall under specific regulations that govern how user-‐generated content should be managed, and the penalties for non-‐compliance have never been higher:
● In 2004, the SEC fined Bank of America $10 million for failing to retain and produce emails in accordance for SEC regulations.
● In 2008, non-‐compliance with FRCP mandates compelled a judge in the United States to award $29 million to a plaintiff in a suit against UBS Warburg.
● In 2010, FINRA fined Piper Jaffray and MetLife Securities a combined $2M+ for email-‐related failures. While the amounts may be modest in the scheme of the financial services industry, we should consider the reputational impact to the companies; particularly after FINRA published press releases referring to “supervisory and reporting violations” as well as “investigations of broker misconduct.”
More Storage, More Problems
While the impact of fines and the publicity they generate should not be understated, they are dwarfed by the costs born by all customers as the storage of user-‐generated data places an ever-‐increasing drain on our budgets, focus, and productivity.
Organizations have tried to address the data explosion by buying more storage. This is illustrated by the fact that EMC and Netapp recently reported year-‐over-‐year growth in storage sales of 33% and 41%, respectively, whereas IT budgets remained nearly flat during the same period. In addition to the capital outlay, each new array consumes focus. New servers need to be installed, networked, configured, maintained, backed up, and monitored by IT. Each consumes valuable data center space, power, and cooling, and many are included in disaster recovery plans, de-‐duplicated, and replicated to a secondary site. In the end, on-‐premise storage systems are considerably more expensive than a line item on a budget would ever reflect.
Beyond the direct and indirect costs of storage systems, maintaining large data stores also impedes the performance of our IT infrastructure and applications. Mail systems buckle under the weight of giant data stores. Network latency increases as backup windows span an ever-‐growing portion of the business day. Users are less productive when their email servers are unresponsive. Businesses risk losing profits when critical applications are slow and unstable!
Archiving is the Answer
At a minimum, information archiving provides customers with a means to address the data explosion in a manner that satisfies regulatory requirements and reduces the burden placed on IT applications such as email. In most cases, archiving provides significant storage and infrastructure cost savings. In some cases, archiving enables IT to redirect focus and resources away from infrastructure and toward value-‐added activities.
It is no surprise then that cost-‐effective information archiving remains at the top of the list of priorities for IT decision makers. Put another way—archiving is no longer a value-‐added service for IT; it is an essential component of the IT portfolio, and it is required to tame skyrocketing storage costs while maintaining compliance. Organizations need to determine the archiving solution that best suits their needs. While there are numerous vendors to choose from, the solutions generally follow one of three models: traditional on-‐ premise archiving, hosted archiving, and cloud-‐powered archiving.
On-‐Premise Archiving
With the traditional on-‐premise model, archiving systems are completely located within a businesses’ data center, and the business maintains responsibility for the installation, configuration, and operation of the archiving system and underlying infrastructure. The archiving software is installed on one or more servers (either as dedicated hosts or virtual machines), and archived data is stored on high-‐volume SAN or direct-‐ attached storage. With on-‐premise systems, customers experience fairly rapid migration of legacy data— attributable in large part to the physical proximity of the archive system to the legacy data store. Historically, on-‐premise archiving software has been offered as a stand-‐alone system, but email vendors (such as Microsoft) have recently integrated basic archiving into their email server products. Capacity management for on-‐premise archiving systems functions similarly to other on-‐premise systems (such as email), where businesses project their own storage and computing needs and procure infrastructure to accommodate needs periodically.
The on-‐premise archiving model was the most popular model for early adopters of archiving solutions (particularly large financial services customers in the early 2000s). Due to the cost and complexity of the systems, which require investments in hardware, software, and storage as well as ongoing operations and support, adoption of this model has been waning as of late. Instead, resource constrained businesses are turning to archiving services that are operated by third-‐parties.
Hosted Archiving
from economies of scale in procurement and operations given that they are serving hundreds or thousands of customers using one infrastructure (often a single data center). In this model, customers focus on activities related to the archiving process and functionality, such as defining retention policies, searching for specific content, and exporting data for discovery.
Many customers are attracted to the hosted model due the fact that it reduces IT complexity and offers cost savings relative to on-‐premise systems. It is also perceived as a fairly low-‐risk evolution of the legacy model in that (unlike cloud-‐powered archiving, discussed below), the archiving system leverages traditional infrastructure technologies. Unfortunately, this also comes at a price, as vendors struggle with many of the same issues related to capacity management, service availability, and large capital expenses that customers faced with their on-‐premise systems.
Cloud-‐Powered Archiving is the Right Answer
Rather than operating their own infrastructure, cloud-‐powered archiving vendors like Sonian, built their applications to operate on top of cloud infrastructure from third parties, such as Amazon or Rackspace. In this model, neither the customer nor the archiving vendor operates physical infrastructure directly. The archiving vendor builds and maintains an archiving system (software layer) that is operated on top of cloud infrastructure. As with the hosted model, the customer focuses exclusively on the archiving process and functionality. Of the three archiving models, the cloud-‐powered approach best capitalizes on the value that can be created through specialization, scale, and elasticity. With this model, the infrastructure vendor, archiving service provider, and businesses are able to focus on their core competencies (operating data centers, developing archiving software, and facilitating business processes, respectively). Likewise, the cloud vendor procures and operates infrastructure at tremendous scale, enabling it to invest in world-‐class architecture while managing capacity and operations more efficiently than archiving vendors and their customers could achieve on their own. This enables cloud-‐powered archiving vendors to offer the lowest prices in the market. Finally, cloud-‐optimized technologies such as ElasticSearch and Chef enable archiving vendors to maximize availability performance based on their customers’ real-‐time processing, bandwidth, and storage requirements.
The Answer -‐ Sonian Cloud-‐Powered Archiving
Sonian provides companies of all sizes with an affordable, scalable, secure, and feature rich hosted email archiving and eDiscovery service. The Sonian Archive Service is an on-‐demand cloud powered service that requires the installation of no hardware or software at your location. Within a matter of minutes, you will be archiving your email automatically from your mail Server (hosted or on-‐premise) to our safe and secure data centers.
All internal and external messages including attachments are automatically copied, stored and indexed. The process is transparent to the end-‐user.
Sonian Cloud-‐Powered Archiving
Some Things are Certain in IT...
Moving forward, growth in the volume of user-‐generated data will only accelerate. The number of
restrictions placed on the management of that data will only increase. Requests (and demands) for data to support litigation, compliance, and business intelligence will continue to rise. IT leaders need to be prepared for the convergence of these trends that, if left unaddressed, will drain the productivity of their teams, increase storage expenses, and put the reputations and financial viability of their organizations at risk.
