Fibre Channel introduces new techniques to attach storage to servers and as a result, it has unique performance issues that affect the overall performance of a server. The purpose of this section is to provide a brief introduction to the
3467
2827
2265
1585 1141
0 1 2 3 4
I/OsPerSecond(Thousands)
RAID-0 WB Maximum Controller Throughput
Requires Maximum Number Of Drives Attached 8K Byte Random 67% Read 33% Write Operations
ServeRAID-3L code v3.0
ServeRAID-2 code v2.4
ServeRAID-3H code v2.7
ServeRAID-3H code v3.5
ServeRAID-3H code v3.6
Always upgrade the firmware on the ServeRAID card to the latest level.
Firmware levels
motivation behind Fibre Channel, to explain how Fibre Channel affects server performance and identify important issues for configuring Fibre Channel for optimal performance.
SCSI has been the standard for server disk attachment for the last ten years.
However, SCSI technology has recently been under stress as it attempts to satisfy the I/O demands of current high-performance 4 and 8-way servers.
Some of the fundamental problems with SCSI are its parallel cable design which limit cable length, transfer speed, and the maximum number of drives that can be attached to the cable. Another significant limitation is that a maximum of two systems can share devices attached to one SCSI bus. This is significant when using SCSI for server clustering configurations.
Fibre Channel was designed to be a transport for both network traffic and an I/O channel for attaching storage. In fact the Fibre Channel specification provides for many protocols such as 802.2, IP (Internet Protocol) and SCSI.
Our discussion in this redbook will be limited to its use for disk storage attachment.
Fibre Channel provides low latency and high throughput capabilities. As a result, Fibre Channel is rapidly becoming the next generation I/O technology used to connect servers and high-speed storage. Fibre Channel addresses many of the shortcomings of SCSI with improvement in the following areas:
• Cable distance
• Bandwidth
• Reliability
• Scaleability
The parallel cable used for Ultra, Ultra2, and Ultra3 SCSI limit cable distances to 25 meters or shorter. This is due to electromagnetic effects impacting signal integrity as cable length increases. Parallel cables such as the type used by SCSI tend to have signal interference problems because of
electromagnetic coupling that occurs between parallel signals traversing the wires.
Serial technologies use fewer signals, typically two or four, compared to as many as 68 for SCSI. Fewer signal lines means less electromagnetic energy emitted and less total signal interference from coupling of the electromagnetic energy into adjacent wires. Lower signal interference allows the serial cable to transfer data at much higher rates than is possible using a parallel connection.
Fibre Channel provides the capability to use either a serial copper or fiber optic link to connect the server with storage devices. Fiber optic technology
allows for storage to be located a maximum distance of up to 10 kilometers away from the attaching server.
The same electromagnetic noise problems that limit SCSI cable length also limit the speed at which data can traverse the SCSI bus. First generation Fibre Channel is capable of transmitting data at 1 Gbit per second in both transmit and receive directions. The most popular version of SCSI, Ultra2 is limited to 80 MBps (bytes) or 640 Mbps (bits).
This difference in performance (1 Gb vs. 0.64 Gb) does not appear to be significant; however, Fibre Channel offers a full-duplex communication path while SCSI is half-duplex. This means that Fibre Channel can achieve up to 2 Gb throughput by transferring data on both send and receive paths at same time. Therefore, the maximum bandwidth of current Fibre Channel
implementations is actually 2 Gb while it is only 640 Mb for Ultra2 SCSI.
However, maximum bandwidth is often touted as an important specification but in actual use, sustainable bandwidth may be much less.
Another significant advantage of Fibre Channel is its ability to connect redundant paths between storage and one or more servers. Redundant Fibre Channel paths improve server availability because cable or connector failures do not cause server down time because storage can be accessed via a redundant path. In addition, both Fibre Channel and SCSI throughput can scale by utilizing multiple channels or buses between the servers and storage.
In addition to a simpler cable scheme, Fibre Channel offers improved scaleability because it offers several very flexible connection topologies.
Basic point-to-point connections can be made between a server and storage devices providing a low-cost simple stand-alone connection. Fibre Channel can also be used in both loop and switch topologies. These topologies increase server-to-storage connection flexibility. The Fibre Channel loop allows up to 127 devices to be configured to share the same Fibre Channel connection. A device can be a server or a storage subsystem. Fibre Channel switch topologies provide the most flexible configuration scheme by
theoretically providing the connection of up to 16 million devices!
The Fibre Channel specification provides many possibilities for how Fibre Channel is configured but we will confine our discussion to the
implementation of the IBM Netfinity Fibre Channel RAID Controller. The IBM Fibre Channel RAID Controller operation can be conceptualized by combining LAN and disk array controller operations.
Figure 55 below illustrates the primary components in the IBM Fibre Channel configuration. The important factors contributing to performance are caused by the RAID controller and storage being attached to the server by a Fibre Channel link. This introduces two factors which contribute to overall Fibre Channel performance. These are:
• The throughput of the Fibre Channel links, shown as the FC bandwidth arrow
• The aggregate throughput of the RAID controller and link combination, shown as the FC-to-disk bandwidth arrow.
Figure 55. IBM Netfinity Fibre Channel RAID organization
In March 2000, IBM introducedNetfinity Fibre Array Storage Technology (FAStT). This new Fibre Channel technology employs second-generation Fibre Channel integrated circuits which greatly improve throughput performance. In addition, device drivers and firmware are optimized to enhance throughput performance. FAStT utilizes the same Fibre Channel protocols used in first generation Netfinity Fibre Channel products but four host connection and four drive connection Fibre Channel links are supported per controller pair to significantly improve total available fibre channel bandwidth.
Up to 60 (6x10) disk drives per RAID controller pair
7.8.1 Fibre Channel performance issues
Let's look at what happens when a read I/O operation is requested to a Fibre Channel subsystem, and the data requested is not located in the RAID controller disk cache:
1. A read command is generated by the Netfinity server and the read command contains the logical block address of the data being requested.
2. The command is transmitted by the Fibre Channel host adapter to the RAID controller over the Fibre Channel link.
3. The RAID controller parses the read command and uses the logical block address to issue the disk read command to the correct drive.
4. The disk drive performs the read operation and returns the data to the RAID controller.
5. The Fibre Channel electronics within the RAID controller format the data into the Fibre Channel protocol format. The data is transferred to the Netfinity server over the Fibre Channel link.
6. Once in the Fibre Channel adapter, the data is transferred over the PCI bus into memory of the Netfinity server.
Of course, a large amount of the detail was left out, but this level of observation is sufficient to understand the most important performance implication of Fibre Channel.
The Fibre Channel link, like most network connections, sustains a data transfer rate that is largely determined by the payload of the frame. Or stated another way, the throughput of Fibre Channel is a function of the disk I/O size being transferred. This is because Fibre Channel frames have a maximum data payload of 2112 bytes. Data transfers for larger data sizes require multiple Fibre Channel frames.
Figure 56 illustrates the effects of disk request size on Fibre Channel throughput. At small disk request sizes such as 2 KB the maximum Fibre Channel throughput is about 20 MBps or about 20% the maximum transfer rate of Fibre Channel. This is critical information as many people think the maximum 1 Gbps throughput is obtained for all operations.
Figure 56. Fibre Channel throughput vs. disk I/O size
Only when the disk I/O size is as large as 64 KB does Fibre Channel reach it's maximum sustainable throughput. In this case the maximum throughput is about 82 MBps. But Fibre Channel is suppose to have one Gigabit
throughput? One Gigabit is roughly 100 MBps (taking into account a 2-bit serial overhead for every byte). The difference between this measured result of 82 MBps and the theoretical maximum throughput of 1 Gbps (100 MBps) can be explained by overhead of command and control bits that accompany each Fibre Channel frame. This is discussed in the following sections.
7.8.1.1 Fibre Channel protocol layers
We can get a better appreciation for this overhead if we take a brief look at the Fibre Channel layers and the Fibre Channel frame composition.
The Fibre Channel specification defines five independent protocol layers.
These layers are structured so that each layer has a specific function to enable reliable communications for all of the protocols supported by Fibre Channel standard.
0.5 1 2 4 8 16 32 64 128 256 512 1024
Transfer Size KBytes 0
10 20 30 40 50 60 70 80 90 100
MB/SecThroughput