Issues That May Drive the Insertion of Digital Optical Communications into High Performance Compute (HPC) Systems

8.1.1 Bandwidth-Distance Product, B*d

There are a number of leading indicators suggesting that high speed electrical I/O signaling may be reaching some of the performance limits of its ultimate high speed digital communication capabilities. Optical communications appears to offer the ability to potentially extend these limits, primarily due to significantly greater signaling bandwidths. For electrical signaling it appears that a "hard limit" for raw data rates may exist in the data rate range between 18 and 44 Gbps, depending upon the system components, the transmission line dielectric

materials and the respective interconnecting line lengths. This limit is at the point where conventional channel compensation methodologies appear to be no longer capable of extending either the line length, the data rate, or both, especially at reasonable power consumption levels.

One key metric used for evaluating digital communications interconnect constructs is known as the Bandwidth-Distance Product (B*d). As noted earlier in Section 5, this metric may be used as a first level approximation to compare electrical and optical communication channels.

An example of the electrical communication link length and data rate limitations for various cost effective interconnect constructs is summarized by Intel Corp [8-1], and illustrated below in Figure 8-1. If we compare the three link constructs shown, the B*d value for electrical

interconnect links is within the range of 500 to 850 Gpbs-cm. Additional B*d limits are reported by other groups within Intel [8-2] in which this limit appears to be near 244 Gbps-cm for shorter length single-board channels. For an FR-4-based PCB construct with longer length and two backplane connectors this value reduces even further to 137 Gbps-cm. For a relative perspective, it is interesting to note that the B*d product for RG-58 Coax cable is reported to be on the order of 30 GHz-cm, or approximately 60 Gbps-cm for NRZ transmissions[8-3].

0.00 0.5 1.0 1.5 2.0 2.5 3.0

10 20 30

Data Rate, Gbps

Power Efficiency, mW/Gbps

40 50 60

17" Refined BP 7" Desktop 8" Flex

From: Young, I., et. al.: Optical I/O Technology for Tera-Scale Computing. From Slide Presentation, IEEE International Solid-State Circuits Conference February 2009.

Figure 8-1: Measured Electrical SerDes Data Rate Limitations and Power Efficiency for Three Representative Electrical Channel Constructs (For Each Data Set the Rightmost Data Point Indicates the

Limiting Data Rate). (40841)

Two other references were examined for the range of values expected for electrical and optical link bandwidth-distance products. They are shown below from research investigations performed by IBM in 2008 in Figure 8-2 and Figure 8-3 [8-4]. Figure 8-2 shows the current state of the art limits for electrical signaling which when converted to B*d is approximately 810 Gbps-cm

020 10 20 30 40

40 60 80

Distance, cm

Maximum Data Rate, Gb/s

100 120

Simulated Maximum Link Data Rate Versus Channel Length

Simulated Hardware With Full Parasitics Simulated Hardware With No IC Parasitics

Source: Pepeljugoski, P., M. Ritter, J. Kash, F. Doany, C. Schow, Y. Kwark, L. Shaw, D. Kam, X. Gu, C. Baks: Comparison of Bandwidth Limits for On-Card Electrical and Optical Interconnects for 100 Gb/s and Beyond. Proceedings of SPIE, Volume 6897, 689701, 2008.

Figure 8-2: Maximum Simulated Electrical SerDes Data Rates. (23485)

The B*d product for optical waveguides is also expected to have a range of values depending upon the optical transmission mode as well as the dispersion characteristics of the waveguide material system in use. A range of typical optical fiber values reported for B*d is listed in Table 8-2 below [8-3].

Single-Mode Multimode Graded-Index Fiber

Multimode Step-Index Fiber Source Edge-Emitting Laser Laser or LED Laser or LED Application Under sea cables Intercity trunks Data Link

Splicing Very difficult Difficult Relatively easy

Cost Medium Most expensive Least expensive

Bandwidth-Distance product (in Gbps-cm

for NRZ)

> 6x10⁵ 6x10⁵≥BW x Dist ≥4x10⁴ <4x10⁴

Table 8-2: Summary of Optical Waveguide Modes and Bandwidth-Distance Product Supported.

Figure 8-3 shows a graph of the B*d limits for multimode optical signaling across the Terabus polymer waveguide embedded within a PCB "hybrid" stackup to be in the range of 1500-3000 Gbps-cm, approaching 6000 Gbps-cm for ideal channel limits.

100

90

80

70

60

50

40

30

20

10

0

Maximum Data Rate, Gb/s

20 40 60 80 100 120 140 160

Distance, cm

EOE With 10 G Terabus Optics EOE With 20 G Terabus Optics 20 G Terabus Optics Only Ideal, Channel Limit Only

Source: Pepeljugoski, P., M. Ritter, J. Kash, F. Doany, C. Schow, Y. Kwark, L. Shaw, D. Kam, X. Gu, C. Baks:

Comparison of Bandwidth Limits for On-Card Electrical and Optical Interconnects for 100 Gb/s and Beyond. Proceedings of SPIE, Volume 6897, 689701, 2008.

Figure 8-3: Maximum Multimode Polymer Optical Waveguide Bandwidth-Distance Product As Indicated By Rightmost Data Point Demonstrated By IBM Research Laboratories. (23631)

To visualize how the B*d product may be expected to affect an idealistic 36-node "all-to-all" HPC system design, we assumed a nominal value of 500 Gbps-cm for an all-electrical link.

Consequently, any links exceeding 500 Gbps-cm (e.g., the longest reaches in the system) may indicate the transition to an optical solution must be made. In this manner we can use the B*d criterion to look at the percentage of optical links required within the system. The result is shown in Figure 8-4 based on processor boards with a 6X6 array of processing elements, using a manhattan routing distance of approximately 10.2 cm (4") for component-to-component pitch.

The summary of the resultant link lengths exhibits a distribution that shows the percentage of interconnecting nets that may be required to be optical links with data rates of 5, 10 and 20 Gbps.

For the slower rate of 5 Gbps, nearly all of the communication links can be satisfied with an

all-Gbps, 83% of the links exceed the 500 Gbps-cm limit and required an optical link implementation.

• •

•

•Uniform array of nodes with 4”Uniform array of nodes with 4”(10.2 cm) pitch in x(10.2 cm) pitch in x--and y-and y-directions directions

•

•Manhattan routing (lines are parallel to either x-Manhattan routing (lines are parallel to either x-axis or yaxis or y--axis)axis)

••Does not include links to switchesDoes not include links to switches

••Processors do not talk to themselves (no loopback)Processors do not talk to themselves (no loopback)

•

•B/W-B/W-distance product is based on 500 distance product is based on 500 GbpsGbps--cm estimate cm estimate

0 20 41 61 81 102

0 50 100 150 200 250

Number of Links

Link Length, cm 10 Gbps 50 cm 38% Optical 20 Gbps

25 cm 83% Optical

5 Gbps 100 cm 0% Optical

Figure 8-4: Summary Showing The Distribution of Node-To-Node Link Lengths For An Idealistic 6X6 Array Of Processing Elements and the Percentage of Links Requiring an Optical Implementation for Three

Different Data Rates (Assuming An Electrical Link B*d imitation of 500 Gbps-cm). (40937)

As silicon IC technology continues to evolve along the Moore’s Law trajectory,

interconnect signaling bandwidth is quickly becoming a system architectural "bottleneck". As discussed earlier in section 5, we also anticipate the transitions onto high speed optical

communication links within HPCs to be gradual. Table 8-3 shows a projection of where these transitions from electrical, to multimode optical, and finally to single-mode optical may occur roughly based upon the assumption that “interconnect performance demand” for B*d will double every 2 years.

2009

Table 8-3: Summary Of Estimated I/O Signaling Data Rate Requirements And The Respective B/W Distance Products Required To Support. (40981)

8.1.2 Optical Interconnect Cost

Many of the current optical (rack-to-rack) applications present in today's HPC designs are dominated by multimode optical links driven with VCSEL sources, and we believe the prime motive behind this current trend is cost. Current cost for DWDM links implemented in single-mode optics are estimated to be in the range of 10-50 $/Gb/s, while the cost of multisingle-mode links with VCSEL sources is significantly lower. Quotations provided to Mayo for low volume VCSEL transceivers have been in the range of 8-16 $/Gb/s for a 10 Gb/s transceiver and 25

$/Gb/s for a 5 Gb/s transceiver configured as a multi-channel active optical cable. These figures are expected to reduce again for high volume manufacturers to 1-5 $/Gb/s [8-5]. The cost advantage may widen even further with projected costs of 0.02 $/Gb/s for multimode optical links implemented in future designs containing multiple dense parallel optical channels in very high volumes [8-6].

Discussions with research and design teams from both IBM and Hewlett Packard [8-7]

have indicated that the cost metric associated with the insertion of any new technology is the single most important metric in determining whether optics will appear for communication links within designs for personal computers, work-stations and servers. A cost-driven “cross-over”

zone for electrical to optical links currently appears to be near the range of 1 $/Gb/s.

8.2 Signal Distribution Characteristics Unique to Optics Applications