XenData Archive Series Software Technical Overview

(1)

XenData Archive Series Software

Technical Overview

Advanced and Video Editions, Version 4.0

December 2006

XenData Archive Series software manages digital assets on data tape and magnetic disk cache on a Windows server. It creates a secure digital archive that is ideal for digital video, medical images, seismic files and a wide range of other engineering and scientific data. The software supports a broad range of tape drives and robotic tape libraries and is highly scalable. It is optimized for archiving and retrieving high volumes of files - from terabytes to multiple petabytes.

2 1. The Basics

XenData Archive Series software manages data tapes and magnetic disk on a Windows file server to create a highly scalable digital archive. The solution may be configured with a wide range of data tape libraries, stand alone data tape drives and magnetic storage. It is ideal for archiving professional digital video, medical images, scientific and engineering data and any application which requires a high performance data repository ranging from a few terabytes to multiple petabytes.

Key Functionality

The software provides the following key functionality:

•

Standard Windows File System Interface The magnetic disk storage and managed tapes appear as a single

standard Windows logical drive which means one or more software applications can write to and read from the archive as though it were a standard disk-based logical drive letter.

•

Strong Data Protection The system combines continuous data protection and automatic tape cartridge replication.

•

Hierarchical Storage Management The solution supports online, nearline and offline files. A file is online when it is available from magnetic disk. Nearline refers to a file that is not on magnetic disk but is available from one or more tape cartridges held in the library or within stand-alone tape drives. Offline refers to a file that is not on magnetic disk but is written to one or more tape cartridges all of which have been exported from the library and/or tape drives.

•

Supports WORM Tape The solution supports both standard rewritable tape and unalterable WORM tape. WORM

tape is particularly useful for compliance applications. Network Configuration

The Windows file server is always configured with a dedicated logical drive on magnetic disk which is managed by the XenData software together with a data tape library and / or data tape drives. A typical system configuration with a single tape library is shown in the diagram below.

XenData Archive Series software is hosted on a server running Microsoft Windows 2000 Server or Server 2003. The equivalent Microsoft NAS operating systems are also supported which are termed “Windows Powered” and “Storage Server 2003”, respectively.

File Storage Policies

(3)

3

The system administrator defines policies that determine where data files are physically stored. A single server may have many different policies, tailored to the needs of the different file types that are being archived. The file management policies are defined via the XenData Management Console which is a Microsoft Management Console (MMC) snap-in. This provides a familiar administration interface which can be easily integrated with other MMC snap-ins. The policies determine whether a file will be written to tape and, if so, a specific group of tapes is allocated. The policies also define whether the tapes will be replicated and the schedule for replica updates. The online retention period is also set by policy and can range from permanent retention on magnetic disk to flushing from disk as soon as the last tape replica has been successfully written at which time the file is available nearline. A typical policy is illustrated below.

With the file management policy illustrated above, a file would be written to both magnetic disk and a specified group of tape cartridges. This immediately provides two copies of the file for high data integrity. The system administrator will have predefined a replication schedule and the number of tape copies for replication. After all copies of the file have been produced, the file may be flushed (deleted) from magnetic disk according to the administrator-defined policy. On reading the file, it is always retrieved from the fastest physical storage device. If it is stored on magnetic disk and tape, the file will be read from disk. If the file is available only on tape, it will be retrieved from tape.

WORM Tape Support

The software supports both rewritable and Write-Once-Read-Many (WORM) tape cartridges. WORM tape provides an unalterable record of the data written to it and provides best practice when authenticity of the digital content is important. XenData Archive Series software supports the simultaneous use of WORM and rewritable media within the library.

The software presents the magnetic disk storage and tape library as a standard rewritable logical drive, even when unalterable WORM tape is being used within the library. The ability to delete and overwrite files is required by many standard applications. Consequently, the rewritable nature of the archive allows standard applications to work with the XenData storage solution without need to write any special interface software. When a file is deleted, it is no longer available using the standard file system interface. Similarly, when a new version of a file is written, only the latest version is available using the standard interface. However, the XenData software includes a browser-based utility, XenData History Explorer, which makes it easy to identify, access and restore all old file versions and deleted files from tape.

Advanced and Video Editions

(4)

4 2. Standard File System Interface

XenData software presents tape and disk as a single rewritable logical drive letter. The use of a standard rewritable file system provides a major benefit; it means that the XenData archive works with standard applications without modification.

Example of how the XenData logical drive letter appears to Windows Explorer. In this case, drive letter E is controlled by the XenData software.

Even when using WORM tape cartridges, files written to the XenData logical drive may be ‘deleted’ or modified as though they are being written to a standard rewritable magnetic disk. However, all versions and all deleted files are retained on tape and can be accessed via a XenData utility.

Microsoft Offline Attribute

XenData Archive Series software uses the standard Windows offline file attribute to identify files that are no longer online on magnetic disk storage. After a file has been successfully written to tape and flushed from magnetic disk, the offline attribute is set. This Microsoft file attribute changes network timeout periods to allow retrieval of files from storage media with long access times. It also changes the appearance of a file within Windows Explorer - a small clock is added to the bottom left of the file icon as shown below.

Standard Microsoft File Security

(5)

5 3. Continuous Data Protection and Tape Cartridge Replication

The XenData storage solution replaces conventional backup because it automatically produces multiple instances of files and metadata, as defined by administrator policies. In case of magnetic disk failure, the complete file system can be easily restored from one of the tape sets.

The multiple instances of a file are created in two steps:

1. Continuous Data Protection Files are always initially written to magnetic disk. Immediately after the file is written to disk, it goes into a queue to be written to the primary tape cartridge. This provides Continuous Data Protection without any backup window! It overcomes the scalability limits of conventional backup.

2. Tape Cartridge Replication Updating of tape cartridge replicas for disaster recovery purposes is automatic and

follows the policies defined by the Administrator. This creates tape cartridge duplicates for off-site retention.

Overcoming the Backup Window Problem

The problem with conventional backup is that it does not scale well above about a terabyte of data. The backup window eventually becomes so large that there is never enough ‘down time’ to complete a full backup. This is illustrated below.

Volume of Data Time for Full Backup

(at 20 MB/s)

1 TB 14 hours

5 TB 70 hours

10 TB 140 hours

50 TB 4 weeks

Taking only incremental backups is not the answer because it leads to a shelf of incremental tapes that soon become unmanageable.

XenData Archive Series software provides continuous data protection, writing to tape immediately after a file is stored on disk. This approach avoids a backup window and the associated scalability limitations. A further advantage of using continuous data protection is that all data is restored unlike conventional backup where changes to the file system since the last backup are lost. Tape Cartridge Replication

(6)

6

When exporting any tape cartridge from the system, including replicas for offsite retention, the location of the tape can be recorded in the XenData Management Console, as illustrated below.

The XenData Management Console allows the Administrator to identify the location of offline tapes

Data Recovery

Data recovery is very easy to accomplish whether it is restoration of the complete file system or the recovery of an individual file that was accidentally deleted. Also, it is easy to test that files have really been written to tape, unlike conventional backup. An individual file can be quickly retrieved from a tape cartridge and opened using the XenData History Explorer utility which provides confidence that the system is functioning correctly.

In case of failure of the RAID, the complete file system can be easily restored from tape using the XenData Management Console, illustrated to the right.

(7)

7 4. Hierarchical Storage Management

Online Retention Policy

The administrator defines rules for online retention periods. After a file has been safely written to both magnetic disk and tape, the file becomes eligible for deletion from disk. This releases the disk space that was occupied by the file but does not delete the file from file system because it is still available from tape. The process of deleting from disk but maintaining file access from tape is termed ‘flushing’. Flushed files have the Microsoft offline attribute set which prevents network timeouts when they are read. If a flushed file is on a tape within an attached tape library or is on a tape within an attached stand-alone tape drive, it is available nearline. When a nearline file is read, it is simply automatically retrieved from tape as though from disk. Of course, the time to start reading the file is longer than if it were on disk.

Configuring Nearline Storage

A high capacity nearline archive can be configured with an attached tape library and relatively smaller capacity magnetic disk which is used as a cache. The administrator sets a policy to flush files from disk immediately after they have been successfully written to tape. By keeping at least one tape from each replicas set within the library, all files are automatically available to the network. This solution is highly scalable and very cost effective..

Offline File Management

After a file has been flushed from magnetic disk and all tapes containing that file have been exported, it becomes offline. The file still appears within the file system. By using the XenData Alert Module, an on-screen and/or e-mail notification will be sent whenever an offline file is accessed. This identifies which file was requested and it’s tape barcode, as shown opposite.

Combining XenData Archive Series software and the XenData Alert Module, provides very effective management of large offline archives.

5. Unalterable WORM Tape

WORM Formats

Most of today’s higher performance tape drives are WORM enabled. The table opposite shows the WORM enabled drives that are supported by XenData software. They are widely used by the leading tape library manufacturers.

All of these drives could be considered as ‘multi-function’ in that they will work with both standard rewritable tapes and with unalterable WORM tape. Using XenData software, the system can be configured to write to only WORM tapes, or to only rewritable tapes or to a combination of both.

File System Audit Trail

The XenData software includes a utility, XenData History Explorer, which provides a file system audit trail. It lists all versions of all files written to WORM tape and their media locations. This includes any deleted and renamed files. It allows retrieval of all file versions and instances directly from tape.

The XenData History Utility means that if you have a file written to WORM tape, it can be retrieved even if the file has been deleted from the file system or is an old version of a file. This avoids theunacceptable situation where you know that a file has been written to WORM media but you do not have the software to access it.

(8)

8 6. Additional Functionality

Additional features of an archive server based on XenData software include:

¾ Multiple Tape Sets - the Administrator can group related files together on the same set of tapes. ¾ Partial File Restore – useful when reading a small clip from a very large video file.

¾ Dynamic Expansion of Tape Sets - the system will dynamically expand tape sets using blank media within a tape library to meet capacity demands.

¾ Cartridge Spanning - the system supports storage of large files across multiple tape cartridges.

¾ Standard Tape Format - open standard TAR file format is used on the tape, allowing the tape cartridges to be read using standard third party utilities. .

¾

Comprehensive diagnostics tightly integrated with the Windows operating system.

In addition to extensive functionality and adherence to industry standards, XenData software has a strong reputation for ease of use.

7. Glossary

Terms used in this paper and other XenData documentation are described in this section.

Activation Code An Activation Code is required to run XenData Archive Series software and this enables the chosen library and drive configuration.

AIT An acronym for 'Advanced Intelligent Tape', a popular mid-range tape format developed by Sony.

API An acronym for 'Application Program Interface'. Most installations of XenData Archive Series software do not use the XenData API. However, a software development kit is available to developers that allows them to use the XenData API to tightly integrate their application with XenData Archive Series software.

Autoloader A term for a tape library which has only one tape drive and cannot be expanded to include more.

Blank Media Set The set of tapes shown in the XenData Management Console which consist of new (unused) tapes or rewritable tapes that have been reformatted by the system administrator.

CIFS An acronym for 'Common Internet File System'. It is the standard protocol used by Windows computers to communicate on a network.

DLTIce Quantum brand name for WORM functionality which is applicable to the SDLT 600 drive and media. Note that the XenData Management Console may be used to format an unused blank SDLT 600 tape as WORM.

Dynamic Disks From Windows 2000 on, Microsoft introduced the option to configure magnetic disk storage as either Dynamic Disks or Basic Disks. The magnetic disk logical drive that is managed by XenData Archive Series software must be configured as a Dynamic Disk.

File Fragmentation The way in which computer systems break large files into smaller, more manageable units for transfer to or from storage devices. Enabling file fragmentation for a File Group allows partial reading of large files and cartridge spanning. File Group A File Group is a collection of files that have the same XenData file management policy and consequently are all treated in the same way by the system. Every file that is handled by the system belongs to exactly one File Group. Files are assigned to a File Group on the basis of their names. This assignment can be based on the name of the directory that contains a file, the name of the file or both. When the 'Save files to tape' option is enabled for a File Group, all files in that File Group will be saved to a designated Volume Set.

Flush Files are flushed (or de-migrated) from a storage device when they are deleted to free space. XenData Archive Series software can be configured to automatically flush files from magnetic disk, once they are securely stored on tape. After flushing, the file is still present in the file system and is available from tape.

FTP An acronym for 'File Transfer Protocol'. FTP is a protocol commonly used to copy files between two computers on the Internet. Both computers must support their respective FTP roles - one must be an FTP client and the other an FTP server. History Explorer See XenData History Explorer.

(9)

9

•

Online with one instance of a file on magnetic disk and, in addition, there may be one or more instances on tape. In

this case the file will be retrieved from disk when accessed over the network.

•

Nearline with at least one instance of a file on tape within a tape library or tape drive and no instance on magnetic disk. Often the file will be written to more than one tape cartridge using XenData’s automatic tape cartridge replication capabilities.

•

Offline with no instance on magnetic disk and one or more instances of a file on tape, all of which have been exported from the tape libraries and ejected from stand-alone drives.

LTO An acronym for 'Linear Tape Open', a popular mid-range tape format which is also known as Ultrium.

MMC An acronym for 'Microsoft Management Console'. The MMC can be used to create, save, and open administrative tools that manage the hardware, software and network components of a Windows system. The XenData Management Console is an example of such a tool.

NAS An acronym for 'Network Attached Storage'. XenData Archive Series software may be installed on Windows based NAS file servers. (The applicable XenData specification defines which NAS operating systems are supported for a particular version and edition of XenData software.)

Nearline See HSM

NTFS Microsoft's advanced file system used to store and manage files on a storage medium. It is the preferred Windows file system when storing files on magnetic disk.

NFS An acronym for 'Network File System'. It is an open standard protocol used by computers to communicate on a network. It was developed by Sun Microsystems and is commonly used by UNIX computers.

Offline See HSM

Offline File Attribute XenData Archive Series software uses the standard Windows offline file attribute to identify files that are no longer online on magnetic disk storage. After a file has been successfully written to tape and is no longer on magnetic disk, the offline attribute is set. This Microsoft file attribute changes network timeout periods to allow retrieval of files from storage media with long access times.

Online See HSM

Petabyte One thousand million megabytes

Primary Tape When new files are written to a Volume Set, they are first written to the tape that is designated as the Primary. For any given Volume Set, the system may change the physical tape that is designated as the Primary. This occurs, for example, when a Primary Tape is exported and replaced with one of its replicas. The Primary Tape is identified within the XenData Management Console.

Quarantined Media Set This is a set of tapes shown in the XenData Management Console containing tape cartridges that have been imported into the library but for some reason cannot currently be used by the system. Typically, this will be because a cartridge has previously been used by a different application (such as a backup application) or because an error occurred while the system was trying to identify the contents of the cartridge.

RAID An acronym for 'Redundant Array of Independent Disks'. RAM An acronym for 'Random Access Memory'.

SAIT An acronym for 'Super Advanced Intelligent Tape', a popular mid-range tape format developed by Sony.

SAN An acronym for 'Storage Area Network'. The magnetic disk logical drive managed by XenData software may be provided from SAN. Note that it must be formatted with NTFS and configured as a dynamic disk.

SDLT An acronym for 'Super Digital Linear Tape', a popular mid-range tape format developed by Quantum. Stand-alone Tape Drive Data tape drive that is not internal to a tape library.

(10)

10

TAR A term derived from 'Tape ARchive'. It is a popular open standard data format optimized for archiving to tape and is the format used by XenData Archive Series software.

Terabyte One million megabytes.

Ultrium A popular mid-range tape format, also known as LTO.

Volume If replication is not enabled, the XenData term 'Volume' refers to an individual tape. If replication is enabled, a Volume consists of a complete set of replica tapes which, when up-to-date, all contain the same data. For example, when replication is enabled and one additional copy of each cartridge has been specified by the administrator, a Volume will consist of two tapes. Volume Set This is a set of one or more Volumes which stores files from designated File Groups. As more data is written to a Volume Set, the initial Volume will eventually become full. At a preset threshold, defined by the administrator, the system will automatically add another Volume by taking the appropriate number of tapes from the blank media set and will extend the Volume Set.

WORM An acronym for 'Write Once Read Many'. WORM tape cannot be reformatted and after data is written to the tape, it cannot be changed. However, further data can be appended. XenData software manages this non-rewritable medium in such a way that files can be ‘deleted’ within the file system in the normal way. However, they are really being hidden from the normal file system interface and can be retrieved using the XenData History Explorer utility. The History Explorer utility can be used to identify and restore all old file versions and all deleted files. When using WORM media, this capability to identify and retrieve all deleted files and old versions is vital when legal compliance is an important issue.

XenData Alert Module This is a companion product which is purchased separately. It provides e-mail and on-screen alerts that are tailored to the needs of archive system operators, systems administrators and IT support personnel. The alerts are derived by filtering and categorizing events recorded in the XenData Event Log.

XenData Event Log This is integrated into the Windows Event Viewer within the Computer Management Console. The XenData Event Log is used for XenData specific messages. If the system is not behaving as expected, the Event Log is the first place that a system administrator should look.

XenData History Explorer This is a XenData utility used to obtain the complete history and status of any file that the system knows about. It can list all available versions of all files, including deleted and renamed files. It allows retrieval of all file versions from magnetic disk or tape.

XenData Management Console This is used to configure all File Group, Volume Set and tape replication options, and to view diagnostic information about the system.

For more information, visit www.xendata.com or contact XenData: USA +1 925 472 6522

XenData Archive Series Software Technical Overview