Optimizing LTO Backup Performance
July 19, 2011
Written by: Ash McCarty Contributors: Cedrick Burton Bob Dawson Vang Nguyen Richard Snook
Table of Contents 1.0 Introduction ... 3 2.0 Host System Configuration ... 4 Host System Random Access Memory (RAM) ... 4 Redundant Array of Independent Disks (RAID) Configuration ... 4 Hard Disk Spindle Speed ... 5 Host Bus Adapter (HBA) Firmware ... 6 Tape Drive Cabling Configuration ... 6 Test Unit Ready ... 8 3.0 LTO Tape Hardware Configuration ... 9 Firmware Updates ... 9 Cleaning the Tape Drive ... 9 Tape Media ... 9 4.0 Data Protection Software Configuration ... 10 Software Updates ... 10 Reported Backup Job Performance ... 10 Block size ... 10 Buffer size ... 11 5.0 Data Set ... 12 Compression Ratio ... 12 Directory Structure ... 12 File Size ... 13 6.0 Conclusion ... 15
1.0 Introduction
Since its inception over ten years ago, LTO (linear tape open) technology has grown to become the dominant tape hardware backup solution. New generations of LTO are introduced approximately every other year with each successive generation vastly increasing the throughput and capacity of the previous. With Dell’s introduction of the PowerVault™ LTO5‐140, the highest performing tape drive on the market today, consumers must consider many software and hardware optimizations to achieve the fastest backup performance. There are four main areas to consider when optimizing throughput on the LTO tape drive: Host System Configuration LTO Tape Hardware Configuration Data Protection Software Configuration Data Set This paper describes these four factors and how they affect total backup performance. Important Note: The figures and performance values in this white paper were compiled using a reference PowerEdge™ servers connected to either a PowerVault™ LTO4‐120 or PowerVault™ LTO5‐140. A mock backup data set at a set compression ratio was used for all tests. Performance increases may vary.2.0 Host System Configuration
Host System Random Access Memory (RAM)
Testing was performed on a variety of Dell PowerEdge™ servers to determine the benefit of increased RAM in the host system and its affect on the performance of a backup operation. The server configurations were tested with 1GB, 2GB, 4GB, 8GB, and 16GB of RAM and the performance and disk I/O was compared for each test run. Based on the test results, server RAM of 1GB or more has no impact on performance increases in backup or restores speeds. Figure 1 ‐ Increasing RAM and its affect on Backup Performance While system RAM may not have a direct affect on the data transfer performance of the tape drive, it is extremely important to note that the amount of system RAM provides additional benefits. Most data protection software applications run a SQL server instance to track information regarding the backup environment, tape drive configuration, and various other components; as your backup environment grows, increased RAM significantly aids the processing of this information. Additionally, increasing the host system RAM is beneficial when using multiple agents and media servers in a backup solution. These resources utilize the additional memory to process backup and restore requests on the master backup server.
Redundant Array of Independent Disks (RAID) Configuration
The backup data set was loaded on various RAID configurations to evaluate how different RAID setups affect the performance of a tape backup solution. The hard drives were configured in a Dell PowerEdge™ server in RAID 1 and RAID 5 configurations. RAID 0 was not tested as it is not recommended for a host system because it does not provide any data redundancy. All RAID setup was performed with hard drives connected to the Dell PERC H200. 0.00 20.00 40.00 60.00 80.00 100.00 120.00 140.00 1GB 2GB 4GB 8GB 16GB Performance (MB/s) Server RAM
PowerEdge T610 and PowerVault LTO5‐140
Reference System
(Average of 64K, 128K, 256K blocksizes)Each test setup was run with Microsoft Windows 2008 loaded on a single hard drive and the mock backup data loaded on separate physical disks in a RAID1 or RAID5 configuration. The data was structured in the following configuration:
Configuration Operating System # of Disk for OS RAID Type # of Disks in RAID
Test Setup 1 Microsoft Windows 2008 1 RAID 1 2
Test Setup 2 Microsoft Windows 2008 1 RAID 5 3
Test Setup 3 Microsoft Windows 2008 1 RAID 5 6
For each test, several backups were performed with mock data to record an average throughput which was measured using the data protection software as well as Microsoft® Windows Performance Monitor. When the backup data was placed on a RAID 5 configuration with three disks versus a RAID 1 configuration with two disks, using a RAID5 with three disks was a performance increase of an average of 18%. Implementing a RAID 5 with the data set on six disks increased performance over the RAID 1 two disk configurations by an average of 28%. Storing the backup data set on a RAID disk configuration will always provide a significant performance increase against a non‐RAID setup, as well as add additional redundancy. A RAID setup will increase performance because it stripes data across multiple disks and allows reading and writing of data faster than a single disk can typically support. Adding more physical disks to the RAID array will also increase performance since it allows the host bust adapter to take advantage of the additional blocks of data striped on the extra physical disks. In this setup, the tests were performed with the operating system on a separate physical disk from the backup data. Generally, although not measured specifically in these tests, this will also contribute to a performance increase and is recommended, as it allows the separate physical disks which contain the backup data to stream at full performance without being hindered by any disk I/O utilization caused by the operating system.
Hard Disk Spindle Speed
Dell offers a variety of hard drive choices on PowerEdge™ servers. For enterprise applications, the disk drive spindle speeds range from 7,200 RPM to 15,000 RPM. There are many benefits to storing your backup data set on the fastest RPM hard drive available. The faster the RPM on the hard disk, the faster the data can be read from the drive's platters, which translates into an overall disk performance increase. The faster the data are read, the sooner the tape drive can process it resulting in a higher throughput to the tape drive.
Testing was completed on a PowerEdge™ server with a data set stored on a 10,000 RPM and a 15,000 RPM hard drive. The results showed an average 20% performance increase when using 15,000 RPM hard drives versus 10,000 RPM hard drives. Figure 2 ‐ Hard Disk Spindle Speed Affect on Performance
Host Bus Adapter (HBA) Firmware
As with the tape drive, it is always important to make sure the HBA has the most up‐to‐date firmware available. For a list of the latest HBA firmware, visit http://support.dell.com, and browse to “Drivers and Downloads”. It is important to take into consideration the type of HBA to which your tape device is connected. Using an HBA that is not supported can result in poor performance or even failure of tape device detection by the host system. Dell certifies its tape devices with a multitude of HBAs to ensure they are properly supported and provide the optimal performance. For a complete list of compatible HBAs, review the Tape Compatibility Matrix at http://www.dell.com/pvmatrix.
Tape Drive Cabling Configuration
When connecting the tape drive or tape automation device to your host system there are several cabling configurations that could inhibit throughput:
Cabling multiple tape drives to a single controller. Due to the amount of processing necessary to manage both devices, the controller’s CPU may inhibit the tape drive from operating at full performance. Always attach tape drives and hard drives on separate controllers.
Cabling a tape drive directly to a RAID controller. This configuration is not valid and may cause your host system to not detect your tape device, or potentially report errors during a backup operation. Never attach a tape drive to a RAID controller. Cabling multiple tape drives to a single host system. While there is no maximum specified number of tape drives per host system, however, it is very important to recognize the effect this will have on each tape backup device’s performance. For example, if you attach four tape drives and all tape drives are streaming data your disk subsystem will be constantly stressed to pump the amount of necessary data required. When using a tape automation library, depending on the interface type there are various ways to maximize performance when cabling the library to the host system: SCSI tape drives in a tape automation device: o For an Ultra 160 SCSI controller: The maximum number of drives recommended to be cabled per SCSI bus is one. o For an Ultra 320 SCSI controller: The maximum number of drives recommended to be cabled per SCSI bus is two. o Cabling more than recommended number of drives per SCSI bus can drastically diminish the performance of the tape drive. Figure 3 ‐ Proper Cabling for two drive SCSI Tape Automation Device (Ultra 320 controller)
Fiber Channel and SAS tape drives in tape automation units: o Since a Fiber or SAS tape device connects directly to a host system or switch (one to one relationship), cabling multiple tape drives per channel is not a concern Figure 4 ‐ Proper Cabling for Fiber Channel / SAS Tape Automation Device For more information about proper cabling procedures for a Dell PowerVault Tape Drive or Automation product, review the Getting Started Guides on http://support.dell.com.
Test Unit Ready
The Microsoft Windows® 2003 operating system sends out a SCSI command known as a Test Unit Ready (TUR) across the SCSI/SAS/FC bus to all attached targets, including tape drives and tape automation devices. The TUR is an informational SCSI protocol command that is used to query attached devices and determine if they are still active on the bus and ready to accept additional commands. This TUR command occurs approximately once every second. This command is usually only sent by data protection software during routine operations when communicating with your tape devices. However, because Windows 2003 sends these TUR commands across the bus as well and usually much more frequently than the data protection software, these can cause performance issues with the tape device. Since the tape device has to process each command it receives, sending it TUR requests takes time and can reduce performance. This is a more noticeable effect on multiple automation libraries attached to a SAN due to the number of host systems flooding the bus with TURs, but it can still cause performance issue on tape drives and automation devices with SAS or SCSI interfaces as well. For more information on TURs and how to disable them, visit http://support.microsoft.com/kb/842411.
3.0 LTO Tape Hardware Configuration
Firmware Updates Dell releases firmware updates and enhancements on a regular basis. These often improve the performance of the tape drive or automation device. Furthermore, the latest firmware versions contain the most recent fixes and will make your LTO tape solution more robust. To get the latest firmware updates for your tape device, visit http://support.dell.com.Cleaning the Tape Drive
Routine maintenance is important to ensure continued optimal drive performance. A tape drive head can become dirty from particles from the tape media as well as from dust in the operating environment. A dirty tape drive head increases error rates that causes the data protection software to perform data retries during the read or write operation; such retries can significantly decrease tape drive performance. Cleaning requests can occur at two different intervals; the first is defined by the amount of usage of the tape drive, while the second cleaning request can occur any time a threshold error rate is reached during a read or write operation. Dell PowerVault™ LTO drives and automation products notify users when cleaning is required. For more detailed information on how to identify when your tape drive requires cleaning, refer to the product user guide at http://support.dell.com. Tape Media Tape Media is a necessary component of a backup solution and is often overlooked as an impediment to performance. Similar to the tape read/write head, as tape media is used it wears. If a tape cartridge is constantly reused and is not part of a media rotation, the potential to induce read or write errors during a backup operation is greatly increased. When the data protection software encounters these read or write errors it will perform retries on the media which decrease the backup performance. Proper media handling and care is essential for optimal performance. Ensuring the media is stored securely and in the proper thermal environment is recommended. Also, dropping the media against a hard surface, even from a short distance, can cause edge damage to occur to the media which can affect performance and if the damage is severe enough, cause potential data loss. The Dell PowerVault™ LTO Media Handbook provides even greater detail on proper media handling procedures for all LTO media and is located at http://support.dell.com/support/edocs/stor‐sys/LTOMedia.
4.0 Data Protection Software Configuration
Dell offers two data protection software solutions which have been certified with Dell PowerVault™ LTO tape hardware: Symantec BackupExec and CommVault Simpana. The performance differences between each data protection software can vary depending on configuration and settings; however, selecting the correct data protection software solution should be based on the features the software offers to fit your data protection goals. For the most updated list of data protection software and operating system compatibility view the Dell PowerVault™ Tape Compatibility Matrix at http://www.dell.com/pvmatrix. Software Updates Each data protection software vendor typically releases hot fixes and service packs several times a year that contain enhancements and fixes. These enhancements can sometimes provide beneficial improvements to the overall performance of the backup solution. To help ensure your software is at the latest version, check the data protection software vendor’s website for information on how to download the latest patches, service packs, or hot fixes.Reported Backup Job Performance
All data protection software reports throughput to the user during the course of the backup/restore operations. However, it is important to understand that these calculations are not always a good indication of true hardware backup performance. These throughput calculations could be off by as much as 10 to 20% and are largely dependent on when the software starts its internal timer (this can range from the submission of the backup job to the start of the first SCSI write command).
Figure 5 ‐ Symantec BackupExec Performance Summary Block size The block size in the data protection software determines how much data is streamed to the device during each write operation and is dependent on the block size the tape drive can support. Performance is largely affected by whether the data set being backed up is optimized for different block sizes. Most data protection software allows the user to modify the block size, and the default is usually 64KB. Increasing the block size can result in a performance increase in certain scenarios depending on a variety of factors. When the LTO4‐120 or LTO5‐140 tape drives were connected to the Dell SAS 6Gbps host bus adapter (HBA) changing the block size in the data protection software did not result in any noticeable performance increase (see Figure 6 below). However, when the same tape drives were connected to the Dell SAS 5/E HBA, changing from the default 64KB block size to 512KB block size resulted in a 43%
increase in performance for the LTO4‐120 tape drive, and a 135% increase in performance for the LTO5‐ 140 tape drive. As demonstrated in these tests different models of host bust adapters may perform differently at varying block sizes depending on how each are optimized. In general, there is no simple rule to follow when adjusting the block size. Perform several backup tests using various block sizes to determine which block size maximizes performance with your data set and HBA. Note: Using a block size of less than 64KB is not recommended as it can significantly decrease performance. Figure 6 – Block Size Adjustments Affect on Performance Buffer size The tape drive must write data at a constant speed, however, the speed at which the data is received from the host can vary. The buffer size determines the size of the tape drive data buffer and is used to temporarily store the data in preparation for writing to the tape. Some data protection software may allow modifying the buffer size. Increasing the buffer size is recommended to improve performance when backing up data sets consisting of many small files. However, for mixed data sets of varying sized files, generally the performance increase is less than 5%. It is important to note that increasing the buffer size often requires more CPU and memory utilization on the host system. As with block size, experiment with buffer size to determine the ideal size based on your typical data sets.
5.0 Data Set
The largest contributor to the overall tape drive performance is the type of data being backed up. There are several factors that can affect performance, including compression ratio of the data, directory structure, and file size. Compression Ratio Each generation of LTO tape technology, whether it is LTO1, LTO2, LTO3, LTO4, or LTO5 has a defined native and 2:1 compressed capacity as well as maximum native and 2:1 compressed theoretical performance value that is defined in the LTO specification. For an LTO5 cartridge, the native and 2:1 compressed capacity is 1500 GB and 3000 GB respectively, while the performance of a LTO5 tape drive is 140 MB/s and 280 MB/s for 2:1 compressed data. Most data in backup environments are not 2:1 compressible and thus will never take advantage of the full 2:1 speed or compression capacity of the LTO device. Every file’s compression ratio is different, and a simple way to gauge the compression of a particular file is to use Winzip to get a general idea of the ratio.Figure 8 ‐ Using Winzip to Approximate Compression Ratios Directory Structure The directory structure of the data is also a contributing factor to performance. Thousands of files buried several directories deep, requires the data protection software to perform additional processing on all the file pointers to each data file in each level. Even though this is a tiny amount of additional data processing per file, thousands of files can affect the throughput of the tape device. Figure 9 – Example of Multi‐level Directory Structure which could affect Performance
File Size Testing was done with a variety of data file sizes on a reference host system and tape backup device to determine the effects of file size on performance. Based on the test results, it was determined that file size has the greatest effect on the backup performance of all variables tested. The test setup consisted of a Dell PowerEdge T610 server with a PowerVault LTO4‐120 tape drive. Using a mock data creation tool, two sets of identical total size backup data were created: Small File Size Test o Number of files: 139,264 files distributed evenly across 4 directories o File size: 250KB each o Total Data Set Size: 33.2 GB (35,641,584,000 bytes) Figure 10 ‐ Performance Monitor Disk I/O measured in MB/s Large File Size Test o Number of files: 340 files distributed evenly across 4 directories o File size: 100MB each o Total Data Set Size: 33.2 GB (35,641,584,000 bytes) Figure 11 ‐ Performance Monitor Disk I/O measured in MB/s The throughput of the backup with the small files was 61% slower than that of a data set consisting of all large file sizes. As can be seen in the hard disk I/O performance monitor data in figure 10, the small files constantly thrash the disk as each small file is read. In addition, due to the number of small files overhead is incurred when the data protection software agent has to open, read, and then close each
file during the backup job. With the set of large files, there is a consistent rate of performance throughout the entire backup set.
