NEW DEVELOPMENTS IN DATA CENTER CABLES & TRANSCEIVERS

(1)

The latest wave in rapid data center

growth is resulting from an increase

in machine-to-machine traffic due

to expansion in server virtualization,

software defined networks (SDNs)

and in cloud computing which is

driving massive demand for

high-capacity data center infrastructures.

High-speed interconnects are playing

an increasingly more important role

in these technologies

25G IS THE NEW 10G; 50G THE NEW 40G;

AND 100G IS AMAZING!

NEW DEVELOPMENTS IN DATA

(2)

Key Takeaways

• Hyperscale and large data centers are driving prices down and speed advancements faster to market dramatically accelerating the entire pace of the industry.

• 25G offers 2.5 times the 10G bandwidth at a small price premium making newer systems an easy upgrade choice for switches, network adapters, cables and transceivers in just about all applications. “25G is the new 10G.”

• Modern data centers have standardized on two form-factors: the single channel SFP and quad channel QSFP.

• DAC cables are used primarily inside the rack; AOCs across the row and multi-mode and single-mode transceivers spanning longer reaches up to 10km across the data center.

• Breakouts of 100G to individual 25G ends are now available for all four interconnect types DACs and AOCs cables and both multi-mode and single-mode transceivers.

• The goal of the plethora of different interconnect technologies is to minimize the costs for every con-nection type while maximizing bandwidth and ROI.

For more information about NVIDIA

®

_Mellanox

®

_LinkX

®

_{cables and transceivers.}

(4)

Introduction

The rapid growth of cellular, video and internet popularity all ends up in the modern data center at the server and storage where the content resides. The path to the server today is via links using DAC and AOC cables and optics based on multi-mode and single-mode transceivers. The shift from 10G to 25G lanes is already well underway with the world’s first 25/100G Ethernet platform consisting of switches, network adapters, cables and transceivers announced by Mellanox in June of 2015.

Today, the industry is transitioning from 10G to 25G line rates as 25G offers 2.5 times the bandwidth but at less then 2X the price premium making newer systems an easy upgrade choice for switches, network adapters, cables and transceivers in just about all applications in hyperscale, enterprise, storage, web 2.0, HPC installations and even telecom data centers are adopting it.

Hyperscale data centers are changing the game by stimulating rapid product advances in speed and form-factors as well as stimulating low prices due to their massive buying practices. The modern data center is being built using 25G line rates in single and four channel versions (100G). On the horizon is changing the signaling scheme from NRZ to PAM4 to offer two data bits per clock instead of one, the transceiver packaging to include 8-channels, and the increased use of single-mode optics that can span the data center all the way to the server. The next hop to 200G and 400G has spawn ~20 different types of optical transceivers in development for every speed, reach, configuration and price point imaginable! With CPU speed advances slowing and becoming more expensive, the only option is to “scale up” – meaning to build a standard rack configuration design using servers, storage, GP-GPUs switches and replicate it in the thousands of racks and build mega data centers. Then interconnect all of it using copper and optical cables and optical transceivers. This is driving the demand for cables and transceivers of all types through the roof.

To simplify things, the data center builders today have standardized on two main form factors or transceiver shells - the single-channel SFP and quad-channel QSFP and four types of interconnects schemes: DAC and AOC cables and optical multi-mode and single-mode transceivers. The legacy 1G and 10GBASE-T interconnect that has been in use for the last 15 –years, has largely been “left in the dust” for the modern high-speed data center.

(5)

| page 5

<< Back to TOC

Why So Many Types of High-speed Interconnects?

High-speed interconnects are all about: 1. Implementing the lowest cost links

2. Achieving the highest net data throughput (i.e. fastest data rate with least amount of data errors, data retransmissions and minimal latencies).

3. Transmitting over various distances

To achieve these goals, various technologies are often used each of which has its own set of benefits and limitations. Data center builders want to build all links with single-mode fiber, duplex LC connectors and single-mode transceivers. Build the fiber into the data center infrastructure once and forget it using single-mode fiber as it does not have reach limitations, then upgrade the transceivers with each new development.

While the fibers and LC connectors are already at the lowest costs, the problem is the single-mode transceivers, which are very complex to build requiring many different material systems, and are both hard to manufacture and expensive. A 3-meter long DAC cable is priced at less than $100 but a 10km reach single-mode transceiver $4,000-$5,000. AOCs and multi-mode transceivers priced in between. As a result, data centers often use an array of different high-speed interconnects matching each interconnect type to specific reach requirements. DAC is the lowest cost but after about 3-5 meters the wire acts like a radio antenna and the signal becomes unrecognizable. AOCs are used from 3 meters to about 30 meters after which installing long cables becomes difficult. More expensive SR, SR4 multi-mode transceivers up to 100 meters after which signal degrades. Parallel single-mode transceivers (PSM4) from 500m-2km. After 500 meters the cost of 8 fibers adds up with each meter, so multiplexing the four channels signals into two single fibers is more economical with CWDM4 for up to 2 km and

LR4 up to 10 km.

In the chart below the longer the reach, different technologies are used. The faster the data rates, the shorter in reach the DAC and multi-mode optics (SR, SR4) becomes while single-mode fiber is largely reach independent.

(6)

DAC – A Cabling Industry Shake Up

CAT-5e cabling and 1GBASE-T have dominated the data center interconnect scene for 15 plus years. However, the transition to 10G Ethernet proved to be technically challenging in both power consumption and cost. That’s when Direct Attach Copper (DAC) cabling, aka Twinax, snuck in and grabbed the lead. Now it has become the preferred, low-cost interconnect for inside server racks, especially for high-speed links at 25G, 50G and 100G in just about all applications in hyperscale, enterprise, storage, and in many HPC installations and is likely to be used for 200G and 400G as well.

Let’s look at the features, advantages and limitations of each of the four major interconnect types used in modern data centers today. First off, there are many different types of high-speed interconnects because high-speed interconnects are expensive to engineer and manufacture. Doing anything at 25 billion times per second is going to be costly. Transceivers may use upwards of 20-30 different material systems and need critical component alignments that are on the order of the width of bacteria! These need to be made in the millions of units and last 20 years! So, different technologies are used for the longer the length of the interconnect span and the faster the line rates. Since high-speed interconnects are often used in the tens or hundreds of thousands in a large data center, costs must be minimized at every connection point.

(7)

| page 7

<< Back to TOC

What is DAC Cabling?

DAC forms a direct electrical connection hence the name, Direct Attach Copper cabling. A DAC is simply two wires where the 1,0 electrical signal is the voltage difference between two wires. A wire pair is used to create one directional lane; so, two pairs creates a single-channel, bi-directional interconnect. Similarly, eight wire pairs form four-channels. Wrap it all up in multiple layers of shielding foil, and solder the wires onto a tiny PCB with an EPROM chip that contains identity data about the protocol, data rate, cable length, etc. Then, put it all in an industry standard plug shell such as SFP or QSFP to create the complete cable with connector ends. While there isn’t much inside DAC cables, there is a lot of design engineering and manufacturing technology that goes into them.

At high signal rates, the wires act like radio antennas. This means the longer the reach and higher the data rate, resulting in requiring more EMI shielding and the cable becomes thicker, usually shorter and more difficult to bend. IEEE and IBTA sets the cable standard specifications for Ethernet and InfiniBand applications. The standard for 10Gb/s signaling supports reaches of 7 meters; the maximum reach for 25Gb/s DACs is usually 3 meters – enough to span up & down server racks. New cable materials are enabling reaches to 5 meters and beyond.

(8)

Low-Price, Low-Power DAC Advantages Stimulate its Popularity

The popularity of DAC can be summed up in two words: low price. Copper cabling is the least expensive way to interconnect high speed systems together. Additionally, the highest volume of links are within racks at reaches of about 2 meters connecting servers and storage. It’s hard to beat the cost of a copper wire, a solder ball and tiny PCB all built on automated machines. More complex technologies such as optical fibers, GaAs VCSEL lasers, SiGe control ICs, InP lasers or Silicon Photonics, which all require sub-micron alignment tolerances, manual labor and a vast assortment of technology piece parts to assemble, cost much more than DACs but support longer reaches.

Besides low price, the other big reason for their enduring popularity is that DAC consumes almost zero power. Several studies show only one Watt saved at the component level (e.g. chip or cable) translates to between 3-to-5 Watts at the facility level. The Wattage multiplier factors in the facility with all the power distribution losses from 100 KV street lines down to 3 Volts and adding in the cooling fans in every one of 54-72 servers in a single rack chassis, plus all the intermediate fans on the way to the rooftop A/C –just to power and cool that one Watt extra. Mellanox Active Optical Cables consume ~ 2.2 Watts; transceivers 2.6 to 4.5 Watts; DACs near zero!

Now, multiply this savings by 100,000 cables and a few dollars saved on each cable on the capital acquisition expenses (Capex) and power consumption operating expenses (Opex) and the costs adds up fast! Large data centers spend upwards of $4 million per month on electric bills and use 25MW! These low-cost, low-power consumption and high-performance capabilities has led “DAC-in-the-Rack” to becoming very popular in hyperscale, enterprise, storage and many HPC systems. DACs create the link between network adapter in servers and storage to top-of-rack switches. Because the number of cables needed in only one rack can be 32-56 or more, even small performance or cost difference becomes very important. Server-to-storage links carry the highest value traffic in the entire data center. All this is especially true in large data centers that deploy tens or hundreds of thousands of cable links.

LinkX Single-Channel SFP DAC

LinkX 4-Channel QSFP DAC

(9)

| page 9

<< Back to TOC

Backwards Compatibility

Mellanox DAC and systems hardware are also line rate backwards compatible. For example, the Ethernet SN2700 32-port 100G switch or 25G/100G ConnectX-5 network adapter card can run at older 1G and 10G as well as newer 25G line rates. Same for InfiniBand equipment with 10G, 14G and 25G line rates. This enables connecting slower or older equipment to newer, faster systems without issues and providing a smooth upgrade path. Another reason for DAC cable popularity is the vast array of configuration options available to create links between old and new hardware in every configuration.

Mellanox LinkX Offers 18 Different DAC Options!

Why so many options? To minimize costs at every connection and embrace the different speeds of various equipment types and both new and older equipment. When systems are fully populated with DAC cables in racks, there is so much cabling that even the LEDs on the servers and switches inside the rack are nearly invisible and blocked out!

Mellanox offers six different physical cabling schemes for interconnecting switches and network adapters to subsystems using SFP and QSFP DAC cables and port adapters.

• SFP-SFP cables • QSFP-QSFP cables

• QSFP-Quad SFP breakout cables • QSFP-Dual QSFP breakout cables • SFP-QSFP adapter cables

(10)

Multiply times two for 10G and 25G line rates which totals 12 different Ethernet DAC options. Add to that six different InfiniBand QSFP-QSFP DAC cables in HDR (4x50G), HDR100 (2x50G) breakout, EDR (4x25G), FDR (4x14G), FDR10 (4x10G) and QDR (4x10G) rates for a total of eighteen different DAC interconnect options. There is even one more if you want to include 14G-based Ethernet that uses 14G FDR InfiniBand ICs to transport the Ethernet protocol – a Mellanox exclusive. Called “VPI”, and unique to Mellanox it enables 4x14G or 56G Ethernet versus the standard IEEE 40G.

At SC’16 in November Mellanox announced the 200Gb/s HDR Mellanox Quantum™_{switches, ConnectX-6}

network adapters and HDR200 QSFP28 DAC cables and a 1:2 splitter cable HDR200-to-Dual HDR100. This brings the total to 18 different ways to create the most cost and performance optimized network links available from Mellanox for InfiniBand and Ethernet protocols.

The graphic below illustrates a data center rack consisting of servers and storage linked via Mellanox 25/50/100G network adapters to Mellanox Top-of-Rack switches using multiple DAC cabling options in single and quad channel versions.

Dual-50G QSFP28 QSFP28100G 25G SFP28 SFP2825G Quad-25G SFP28 QSFP28100G 100G QSFP28 QSFP28100G

QSFP-SFP Port Adapters & Cable

Straight Cables

Breakout/Splitter Cables

(11)

| page 11

<< Back to TOC

QSFP-to-SFP Adapter Enables Many Options

Sometimes the simplest products can solve big problems. The DynamiX QSA is a mechanical port adapter that enables plugging a single-channel SFP device into a quad-channel QSFP port. Available in a 10G and25G versions, only one channel passes through the QSFP. Shown below, the DynamiX QSA enables all kinds of interconnect devices from 1G, 10G, to 25G in DAC and AOC cables including multi-mode and single-mode SFP-based transceivers spanning reaches from 0.5 meters to 10km. All of these options can plug into a QSFP28 port in a switch or network adapter.

(12)

BER – Designed for High-Performance Computing (HPC)

It’s been said that nearly anyone can build a 10G DAC cable. But not everyone can build one that works perfectly at blazing fast speeds of 25G that operate for many years under high temperatures, under all conditions found in modern data centers and one that does not induce bit errors into the data streams. DAC cables are most frequently used to link high value data from compute servers so maintaining the lowest error interconnect is critically important.

All Mellanox DAC cables are designed to HPC InfiniBand supercomputer Bit Error Ratio (BER) standards (even our Ethernet DACs). which requires a BER rating of one-bit error in 10^15th bits transmitted (expressed as 1E-15). The IEEE Ethernet industry standard is BER of 1E-12 which is one-bit error in 10^12th bits transmitted or about 1,000 more errors than Mellanox standard DAC cables. All Mellanox DAC cables are tested to BER of 1E-15. Which cable would you choose to send your electronic pay check over?

Too many bit errors from poor quality DAC cables means data packets get dropped and the data must be retransmitted making 100G more like 85G. And most operators won’t even know it!

(13)

| page 13

<< Back to TOC

“Just Use FEC to Clean it up!”

Not so fast! We’ve heard many data center operators say, “We’ll just use FEC to clean up the errors”. In the server rack, the use of Forward Error Correction circuits (FEC) is not recommended at reaches <2 meters per the latest IEEE spec at 25G line rates and 2 meters is the most common reach for DAC for linking high-value servers! FEC adds about 120ns delays each way. For server uplinks where all the traffic is, this delay can really slow things down. FEC can detect and correct only so many errors before it becomes overloaded and forces a packet retransmit. Server uplinks are the most important links to maintain error free as they account for 65 percent of the total hardware costs and most of the volume of data traffic in a data center as it is where all the data is processed. So, keeping these links efficient and error free is very important to maintaining high throughput and maintaining optimal ROI.

More Value – Extending the Reach Past 3 Meters

Mellanox DAC cables typically can reach significantly further than competitor’s DAC cables which often just barely achieve the IEEE standards of 3 meters and at a BER 1E-12. Without using FEC on the host, Mellanox DAC cables can reach as far as 5 meters (16 feet) which is enough to span 3-4 server racks. Using competitor cables with 3 meter limitations, data center operators have to resort to more expensive AOCs or optical transceivers for reaches past 3 meters proving that going cheap is usually more expensive.

Note: The use of FEC, what types of FEC, cable thickness, and lengths are currently hotly contested subjects in the 25G industry and the IEEE with no firm decisions yet – so stay tuned!

(14)

Zero Latency Delays

InfiniBand market requirements are much more stringent about signal quality than Ethernet markets as InfiniBand systems are all about minimizing latency delays. So, InfiniBand markets avoid the use of FEC, which can cost 120ns each way to clean up data errors. Since DAC cables have no electronics or opto-to-electronic conversion in the data path, as do optical devices, DAC latency delays are near zero. In big data centers with complex cabling configurations, the latency delay for each hop can rapidly add up with all the various interconnects that data must pass through so minimizing it is of key importance to data center operators today.

Learn from DAC Disasters

Many “inexpensive” DAC cables typically use shoddy manufacturing techniques, sample testing (maybe 1 in 10 cables tested) and less electrical shielding in the cable to save costs. Often the signal is sitting right on the edge of failure with little margin for error. Small changes in the cable can push the signal over the edge and cause a loss of the link or worse induce random intermittent data error bursts. This results in many installations having the dreaded, “DAC Disaster”. This is where going “cheap” becomes “really expensive” when factoring in system down-time and chasing down intermittent signal losses and drops from low-quality cables. Link drops have even occurred by simply moving a cable a few inches to see the port number on the switch. Signaling is at the near margin limit, the shielding at the bend in the cable opens, signal squirts out, and the link drops. Just try to diagnose that problem! Some installations had to completely replace the DAC cabling because of “going cheap”.

Mellanox DAC cables offer BER 1E-15 without the host FEC enabled (vs IEEE standard of 1E-12 with host FEC enabled) means there is a lot of signaling margin left to absorb signal losses and random or burst-mode noise.

The graphic below shows all the different Mellanox cabling options for linking rack systems to Top-of-Rack switches for both InfiniBand and Ethernet protocols. InfiniBand uses primarily a 4-channel link and only recently with HDR100, a 1:2 splitter cable. DAC cables can also link various subsystems to other subsystems directly without going through a switch as well. Shown in 25G line rates, the switches, network adapters and DAC cables are all available in 10G line rates too.

DAC Uses in Ethernet and InfiniBand

“Some things

in life you don’t

go cheap on –

eye surgeons,

parachutes and

(15)

| page 15

<< Back to TOC

Mellanox DAC Manufacturing

Most DAC manufacturers only build DAC cables. Mellanox designs and manufactures all its own switch systems, network adapters, DAC and AOC cables and optical transceivers. Not only the systems but also the ICs that go into the switches, network adapters, and transceivers. This vertical integration, “end-to-end” approach ensures everything is designed to work optimally together. The vertical integration and manufacturing enables Mellanox to offer and unparalleled “out-of-the-box” quality experience for our customers. Mellanox has multiple facilities in multiple geographic regions that manufactures DAC cables in high volumes.

At Mellanox, so called, “PLUG & PLAY” means “PLUG-IN AND WALK-AWAY”;

not the usual Plug & Play-All-Day needed to get things to work.

Every Cable Tested in Real Systems

As Mellanox is also a switch and network adapter systems company, we test every DAC cable in real switching and adapter systems 32-48 at a time for extended times under raised temperature conditions found in actual systems deployment. This is unlike most competitors who typically test one at a time (or sample testing) on a technician’s bench for a few minutes using manual labor and expensive test equipment. To save costs, many manufactures don’t even test every cable and sample test one in a lot of 5 or 10 leaving the final test burden on the buyer. Mellanox test every DAC cable to a BER of 1E-15 –one thousand of times better than competing Ethernet cable suppliers. So, there is a lot of spare signal margin in Mellanox cables rather than, “just barely qualifying and operating the edge” as many competitor cables do.

Key DAC Cable features:

• Lowest cost, high-speed

interconnect solution

• Acts as a plug & play cable

• Available in multiple

configurations for every

application from 0.5 to 7

meters

• Point-to-point links to connect

racks together and breakouts

cables link switches to servers

and storage systems

• QSA adapter enables linking

single channel devices to quad

channel ports in switches and

network adapters

(16)

DAC Cables - Why Choose Mellanox?

Some buyers attempt to shave a few dollars building “Frankenstein” systems from multiple vendor’s equipment, but they often end up paying big time in qualification, maintenance and reliability. At 1G this was easy to do but at 25/50/100G it is another matter. In e-commerce applications, even one minute of down time can be very costly.

The combination of high-quality cable materials, Mellanox designed and manufactured cables using real systems testing and at a minimum standard of 1E-15 BER makes Mellanox LinkX cables a preferred choice in high-speed, critical systems applications and at blazing 25G line rates that includes just about all applications! If you can find any other way to interconnect switches and network adapters using DAC cables – we’d like to hear about it! DAC cables are a tool in the networking tool kit and it’s important to understand use advantages and limitations.

Why Mellanox? More cabling options, exceptionally low bit error and low-latency ratings, tested in real systems, designed and manufactured by Mellanox.

(17)

| page 17

<< Back to TOC

AOCs - Active Optical Cables

Active Optical Cables (AOCs) are the lowest priced optical links available. AOCs are generally used at reaches of 5-30 meters but can reach up to 100-meters. They are widely used in HPCs and more recently became popular in hyperscale, enterprise and storage systems as a high-speed, plug & play solution with longer reaches than DAC cables.

What is an AOC?

Optical transceivers convert electrical data signals into blinking laser light which is then transmitted over an optical fiber. Optical transceivers have a detachable optical connector to disconnect the fiber from the transceiver. AOCs bond the fiber connection inside the transceiver end, creating a complete cable assembly much like a DAC cable, only with a 100-meter reach capability. AOCs main benefit is the very long reach of optical technology, while acting like a simple, “plug & play” copper cable. A complete 2 transceiver AOC including fiber cable assembly is usually priced slightly above the price of only a single connectorized transceiver for several reasons we’ll show below!

What are AOC Features and Advantages?

Compared to less expensive DAC cables, AOCs offer: • Longer reach capability than DAC 3-7 meter limits • 3m – 100-meters multi-mode technology

• Lower weight, thinner cable and tighter bend radius enabling more flexible configurations, increased airflow for cooling and easier system maintenance

Key AOC Cable features:

• Lowest cost optical solution

• Acts like a plug and play

cable but offers optical

long reach features without

optical connector hassles

• Available in point-to-point

configurations to link racks

together and breakout

links individual servers and

storage subsystems

• Lower power consumption

& lower maintenance than

optical transceivers

(18)

Compared to more expensive optical transceivers, AOCs offer:

• Dramatically lower priced solution than two optical transceivers and connectorized fiber based links • Lower power consumption at 2.2 Watts per end versus up to 2.6 to 4.5 Watts for optical transceivers

(4-channel)

• Lower operational and maintenance cost with no optical connector to clean or repair.

The photo above shows the use of thousands of AOCs in a HPC super computer at the University of Texas. The AOCs are precisely manufactured with all eight fibers cut to exact lengths to minimize the time skew between each of the four channels within the AOC cable end. This is to enable each of the four individual signal pulses to arrive at the transceiver end at exactly the same time. Believe it or not, the speed of light delay traveling at 130,000 miles per second in the fiber is a “significant issue” over a 10 meter AOC in an HPC supercomputer. High-speed computing is all about minimizing latency delays between critical components.

How AOCs Differ from Other Interconnect Solutions

Permanently attaching the fibers is a seemingly simple change but yields a surprisingly large number of technical benefits and cost advantages; enough to create an entirely new category of interconnect products. Since the optics is contained inside the cable, designers do not have to comply to IEEE or IBTA industry standards for transceiver interoperability with other vendors. This gives designers complete freedom to can pick and choose the lowest cost, best performing technologies since the cable is a closed system and has a predefined cable length. All of this results in a dramatic cost and price reductions. Here are some of the results this simple change enables:

1. Lowest priced optical interconnect available – near half the price of a single optical transceiver – much more than just the cost of deleting the optical connectors.

2. Plug & play: Ease-of-use “cable” features – like DAC cables only with optical long reaches 3. Long reach: Up to 100 meters

(19)

| page 19

<< Back to TOC

4. Lowest optical power consumption per end – significantly lower than connectorized transceivers – saves operating expenses in power consumption and cooling

5. No optical connectors to clean and maintain – saves operating expenses and increases reliability 6. Optical isolation – isolates electrical systems from ground loops as with copper DAC cables – a

reli-ability advantage

Achieving Lower Product Cost

1. Testing Costs: Optical testing accounts for 40-50 percent of the total cost of manufacturing a trans-ceiver. AOCs can be tested in switch system as an electrical test. Cable is plugged in; test patterns and data run; come back later and look at the results. If good, ship! If not, scrap. Optical transceivers, on the other hand, are much more complex, requiring $500,000 of optical test equipment per station, a very experienced (e.g. expensive) test technician, and a lot of manual time on the test bench. AOCs do away with all of this since the testing is only in the electrical domain. Mellanox uses its “scratch & dent” switches to test AOCs and is one way we achieve a bit error ratio (BER) feature of 1E-15 versus 1E-12 IEEE standard.

2. Design Freedom: Since the optics is contained inside the AOC cable, designers can utilize the lowest cost materials and transceiver designs. VCSELs that don’t qualify for transceivers can be reused and low cost, short reach (orange colored) OM2 fiber can be used for <20 meter reaches saving more expensive OM3 and OM4 fiber for longer reaches.

3. Freedom from Industry Standards: AOCs must comply to the IEEE, IBTA and SFF industry standards for the electrical, mechanical, and thermal requirements but he hardest part are the optical requirements. Since the optics are contained inside the cable, they do not have to meet any standards hence allows a lot more design and material use freedom and eliminates the costly optical testing.

AOCs Offer Lower Operational “Hidden Costs”

1. AOCs do not have optical connectors to manually clean every time they are removed as a single speck of dust inside the connector can completely block the 50-um or 9-um diameter fiber light transmission area. In a transceiver link, there are two fiber ends and two transceiver ends to clean. Besides, the personnel cost the connector cleaners can cost upwards of $250 each.

2. AOCs don’t use MPO or LC optical connectors which, in crowded racks, can be dropped and the fiber end scratched rendering them useless.

3. Optical connectors can channel an electrical static charge that builds up on a long plastic cable and can destroy the sensitive optical transceivers electronics when connected.

4. AOC is a “plug and play” cable solution rather than a “plug, assemble and clean” solution as with opti-cal transceivers. Optiopti-cal transceivers, fibers and connectors also have many different and complicated product variances. All these must all be exactly matched to the specific transceiver used and spares kept and a technician trained in the specifications.

5. AOC cables have a short bend radius and much thinner able thickness than most DAC cables. This makes them easier to deploy and frees up a lot of space for increased air flow cooling in crowded systems.

6. MPO optical connectors are known to insert half way into the transceiver and look fine to the techni-cian later creating problems and maintenance issues.

(20)

7. Lastly, there are big operational savings in power consumption costs. One Watt saved at the compo-nent level translates to 3-5 Watts at the data center facility level. This is when all the chassis, row, room and facility fans and air conditioning equipment is included along with the electrical power to drive them – not counting the repair and maintenance! AOCs are less complex than optical transceiv-ers and offer lower power consumption. Mellanox designs its own AOC ICs, so can offer incredibly low power consumption ratings of 2.2 Watts per end compared to competitors at 2.5-3W.

How Are AOCs used in Modern Data Centers?

While AOCs reaches can extend to the limits of the optical technology used (100-200 meters), installing a long 100-meter (328 foot) cable, complete with an expensive transceiver end, is difficult in crowded data center racks so the average reach typically used is between 3-30 meters. Only one “oops” per cable allowed. Damaging the cable means replacing it as it cannot be repaired in the field. AOCs are typically deployed in open access areas such as within racks or in open cable trays for this reason.

Mellanox’s InfiniBand AOCs started out in about 2005 with DDR (4 x 5Gb/s) for use in the Top10 HPCs and quickly became the preferred solution for the large InfiniBand HPCs in the Top100 in which Mellanox is the market leader. Today, AOC use is the norm for QDR, FDR and EDR and in 2017 the newly announced LinkX 200G HDR announced at SC’16 event in November.

The power and cost savings caught the eye of the Ethernet hyper scale and enterprise data center builders and has since become a popular way to link Top-of-Rack switches upwards to aggregation layer switches such as End-of-Row and leaf switches. Several hyperscale companies have publically stated their preferred use of AOCs for linking Top-of-Rack switches. Additionally, single channel (SFP) AOCs have become very popular in high-speed, NVMe storage subsystems. Some hyperscale builders often run 10Gb/s or 25Gb/s AOCs from a Top-of-Rack switch to subsystems 10-30 meters away – much further that 3-7 meter reach of DAC cables.

On the down side, AOCs have an expensive transceiver end which is difficult to install in crowded areas over long reaches, hence the average use is 10-30 meters. AOCs are typically deployed where there is easy access to cable trays or open areas. When an AOC fails, most operators simply abandon it in place and run another AOC.

Why Choose Mellanox AOCs?

Mellanox not only designs its own AOC control ICs such as a TIA/Laser Driver but the company designs, manufactures and tests every AOC cable as well. The vertical integration of components enables maximizing the signal margin at every point.

Mellanox was one of the first adopters of using AOCs in high-speed HPC supercomputers and has a long history and experience in AOC deployment and manufacturing.

(21)

| page 21

<< Back to TOC

Above is an example of how AOCs are typically used inside systems racks to link subsystems together and between switches and across systems in rows.

Below is a more detailed view of Ethernet configurations showing Mellanox’s LinkX 10Gb/s and 25Gb/s based AOCs, Mellanox Spectrum®_{switches and ConnectX-3, 4, 5 QSFP and SFP network adapters.}

Additionally, Mellanox recently announced two new AOC that break out to 25G and 50G ends. A 100G QSFP28 AOC split out into four 25G SFP28s ends and another cable with a dual 50G QSFP28 break out.

(22)

Short Reach Optics in Modern Data Centers

Short Reach (SR) multi-mode optics are the lowest priced optical interconnects available today that use detachable optical connectors to separate the transceiver from the optical fibers. Although AOCs and SR transceivers both support 100m reaches, AOCs are much less expensive to manufacture but due to installations difficulties generally used at less than 20-30 meters reaches. However, SR optics is frequently used up to 80-100 meter reaches. The key advantage of SR transceivers is the transceiver can be separated from the optical fiber infrastructure that may be permanently installed in structured cabling pipes, under computer room floors, or even between multiple floors.

Short reach optics are not new and has a long history of different fibers, connectors and transceiver types at different data rates creating a blizzard of different parts and complex, specific features to each. But modern data centers have simplified things and zeroed in on the single-channel SFP with the duplex 2-fiber LC optical connector and the four-channel QSFP with the 8-fiber MPO connector. Mellanox offers both these configurations in both as 10G and 25G line rates and quad versions at 40G and 100G. Multi-mode fibers have a large 50-um diameter light carrying fiber core. This makes SR transceivers easier and less expensive to manufacture compared to single-mode optics with tiny 9-um fiber cores which are difficult and expensive to build with. For the low-cost reason, multi-mode, short reach optics are very popular in modern hyperscale, enterprise and storage data center applications as an optical solution. Approximately 70-80 percent of the links in a data center are at reaches less than 60 meters traveling up and down rows of racks and well within the 100-meter reach of short reach transceivers.

(23)

| page 23

<< Back to TOC

VCSEL Multi-mode Lasers & Fibers

Short reach transceivers use a laser that is built on a gallium arsenide (GaAs) semiconductor wafer and constructed perpendicular to the surface of the wafer. When excited by electrons, GaAs emits light at 850nm wavelength and is channeled into a vertical cavity on the wafer surface where it resonates and becomes laser light. Hence the name, Vertical Cavity, Surface Emitting Laser or VCSEL.

Multi-mode optics employ a large core

diameter, 50-um, optical fiber that is easy to interface to VCSEL lasers and detectors, so the costs are much lower than single-mode optics with a tiny 9-um core diameter fiber that are difficult to align. But the SR laser pulse tends to scatter into multiple transmission paths or “modes” (hence the name multi-mode). The scattered pulse in large diameter fibers becomes unrecognizable after about 100 meters so the IEEE standards body sets the limit at 100 meters, assuming four connectors in the run. Multi-mode can reach to 400m, but requires specialized lasers, fibers and connectors and are priced near that of single-mode transceivers. For reaches longer than 100m, single-mode optics is generally used.

Connectorized Optics – A Key Feature

Many data centers have structured cabling where the fiber infrastructure is fixed and installed in cabling pipes, under raised floors and integrated into optical patch panels used to manually reconfigure the fiber run end points. Sometimes, fibers run to other system rows, rooms, floors, or even other buildings necessitating the ability to disconnect the fibers from the transceivers installed in the systems. This is something that DAC and AOCs integrated cables cannot do as the wires or fibers are integrated into the plug or transceiver end. Multi-mode optics uses the 2-fiber LC and the 8-fiber MPO optical connectors.

Point-to-Point Applications

ToR-to-Leaf/spine EOR switches

One of the main applications for SR and SR4 transceivers are to link Top-of-Rack (ToR) switches to other parts of the network such as aggregation switches, middle and end-of-row switches, and to leafs in a leaf-spine network. These are typically used as high bandwidth busses that are four-channel SR4s at 40G or 100Gb/s bandwidths. Single channel SR 10G and 25G transceivers can be used to link server and storage systems to a ToR switch within a rack or adjacent racks. Since the number of links is very high the closer to the server one gets, low cost optics is important and multi-mode optics is well suited to these applications where the reaches are relatively short spanning within a rack or along a single row. While several enormous hyperscale operators have made a lot of noise in the press around moving to single-mode fiber, many big hyperscale and enterprise installation still operate as groups of small system clusters where all the systems are well within the reach of 100m mode fiber. Interestingly multi-mode fiber is about three times more expensive than single-multi-mode fiber, the single-multi-mode transceivers are 50 percent to 10X more expensive than multi-mode transceivers. Single-mode transceivers are difficult to build but offer reaches up to 10Km vs only 100m of multi-mode. Single-mode fiber is the mainstay of the telecom industry linking cities and countries together hence is made in thousands of mile spools.

Key SR/SR4 Transceiver features:

• Connectorized optics –fibers

can be disconnected from the

optical transceiver

• Point-to-point and breakouts

– meaning a SR4 transceiver

can operate as a single,

four channel link -or- as four

separate channels with links

to individual subsystems.

• Available in 10G & 25G line

(24)

Breakout Applications

ToR QSFP Breakouts to SFP Servers & Storage

Linking Top-of-Rack switches down to servers and storage subsystems within the same rack is another popular use for SR and SR4 optics. In the past, SR4 transceivers only transferred at 4-channels at a time to another SR4 in a switch-to- switch application. Newer transceiver models can split the four into individual single-channels that can be connected to different systems and operate independently. This is important when the link reach needed is greater than the 3-meter capability of DAC copper cables and perhaps spanning more than one rack. The passive fiber break out cable has a single 4-channel MPO on one end connecting to the SR4 transceiver and four Duplex LC optical connectors on the other end connecting to four separate SFP transceivers each with their own 100m fiber run.

Increased Airflow; Tighter Bend Radii; Easier Rack Maintenance

Breakout optics also provides a linking alternative in the rack that is much smaller than DAC cables. This is due to the tiny fiber diameters compared to DAC cabling. All the optical cables together supporting a 32-port switch has a diameter less than 2 cm (3/4-inch) compared to about 100-125cm (4-5 inches) with copper DAC cables.

Mellanox sells short reach multi-mode optics in 10G and 25G line rates and single and four-channel configurations enabling 10G to 100G of link bandwidth. These are available in SFP and QSFP connector form-factors that use LC and MPO optical connector respectively.

100G QSFP28 Breakouts to Dual 50G QSFP28 and Quad 25G SFP28s Breakouts Using 8-fiber MPO and 2-fiber LC Optical Connectors

Thirty-two optical fiber cables would blow off the table with a sneeze

but 32 DAC cables could qualify as exercise equipment!

(25)

| page 25

<< Back to TOC

Can 50G Cost Less Than 40G?

Answer: Yes, use breakout cables or transceivers with splitter fibers and split the ports on a 100G Mellanox SN2700 or SN2100 ToR switch.

Top-of-Rack switches such as the Mellanox SN2700 supports 32 QSFP28 ports of 100G each. The SN2100 is half the width and offers 16-ports of 100G. By using DAC splitter, AOC splitters, or transceivers and splitter fiber cables, ports in the switches can be split into a total of 64 50G ports. In this way, one 32-port or 16-port Mellanox SN2700/2100 switch makes 50Gb/s less expensive overall than a 32-port 40Gb/s switch. With this approach, you end up with 64-ports instead of 32 and each run 20% faster! Additionally, it provides an upgrade path to 100G by simply changing the network adapters and 50G cables or transceivers to 100Gb/s.

64-Ports of 50G Using Mellanox Ethernet Switches & Breakout DAC and AOC Cables and Transceivers

32 switch ports at 40Gb/s with no upgrade path or 64 ports at 50Gb/s using break out cables with

an upgrade path to 100G. 2.5X the bandwidth at less than 30%-50% price premium – you do the math!

32-ports of 100G Switch or 64 ports 50G

SN2700 32-Port 100G Switch SN2100 16-Port 100G Switch 16-ports of 100G Switch

or 32 ports 50G 16-ports of 100G Switch or 32 ports 50G

Similarly, two 50Gb/s links can be created from one 100Gb/s using an MPO breakout cable with two MPOs connected to 50Gb/s SR2 or SR4 transceivers using only two channels each (2x25G) as shown in the following figure.

(26)

Typical Data Center Short Reach Transceiver Applications

Short reach transceivers at bandwidths of 25G, 40G, 50G, 100G have many uses in modern data centers and are very popular. Unlike AOCs, short reach transceivers have optical connectors that can be disconnected from the transceivers and routed into the data center infrastructure.

The figure below illustrates using short reach optics with fiber splitter cables for inside and between rack applications connecting subsystems where the reach may be beyond that of 3-5 meter DAC cables or where increased air flow is needed using the thinner fiber cables. Additionally, short reach transceivers are often used along rows to link to storage and server subsystems. Lastly, to link Top-of-Rack switches to End-Of-Row and aggregations switches up to 100 meters in system rows or clusters.. Extended reach SR4 transceivers can increase the reach to 300 and 400 meters depending on the fibers (OM4), high-grade MPO optical connectors and perhaps a lower bit error rate tolerance.

(27)

| page 27

<< Back to TOC

Single-mode Optics Increasing Popularity

What is Single-mode Optics?

Single-mode optics is all about long reaches enabled by using single-mode fiber – the same fiber used by the telecom industry to send Internet data between cites and across oceans. Traditionally, single-mode transceivers have been exceptionally expensive but Hyperscale builders began ordering these in huge volumes are driving down the cost.

Single-mode transceivers use various technologies to achieve price and reach targets. Lasers are made using Indium Phosphide (InP) and emit in the 1310nm. Lasers can be directly modulated via turning on and off electrical currents (called DMLs) or use continuous wave lasers and modulating the light externally (called EAs or EMLs). Lasers, modulators, waveguides, multiplexers, detectors and other elements all have to be integrated or precision aligned to “bacteria level” tolerances of hundreds of nano-meters. Difficult to do at all, harder and make a million units reliably and be attractively priced.

Single-mode Transceiver Advantages

• Enables using low-cost, long fiber reaches up to 10 km

• Makes the type of fiber and reach used anywhere in the data center a non-issue • Can support hundreds of wavelengths in a single fiber

• Is largely data rate speed agnostic

The main advantage of single-mode transceivers is it enables using inexpensive single-mode fiber that has a very long reach capability. Recently, as the transceiver costs drop, it is also being used for in-rack breakout applications like DAC, AOCs, and multi-mode transceivers configurations mentioned above. The long reach capability makes it one of the most flexible interconnect types as it makes reaches in a data center a “non-issue”. By employing a tiny 9-um core optical fiber, the data signal pulse is recognizable at the receiver over an astounding 100 km reach! Which is why the telecom industry uses single-mode fiber to connect cities and countries together.

Large data centers are moving to deploying single-mode fiber as fast as they can to get rid of the numerous multimode fiber short reach hops and fiber/connector/switch complexities required to send data past 100m. These are costly, expensive to maintain, and must be constantly upgraded with each line rate speed advances. With speed advances happening every 2-3 years instead of 5-8, data center operators are looking to eliminate these costs. Multi-mode fiber is tuned to specific data rates and bandwidths. Every increase in data rates means the multi-mode fiber reach becomes shorter or fiber needs to be upgraded (OM2, OM3, OM4 now OM5). Add to this the complexity of many different types of MPO connectors with different quality ratings, key-up/key-down, polarity, male/female, crossover issues. Large data centers have very complex networks inside that create optical losses as well as having enormous physical data centers as long as 1 km and also need to connect to the metro area network using 10km transceivers. Lower costs and long reach is what is driving the popularity of single-mode optics in data centers.

To go 200m, four SR4 transceivers would be needed with a network switch in between. With single-mode optics just 2 transceivers and that could send data 10 km. While single-mode transceivers are more expensive than multi-mode transceivers, when considering a long link, the total cost of fiber, connectors, transceivers, and an extra network switch is less with single-mode optics.

Hyperscale data centers want to build a huge data center infrastructures once (some costing $1 billion) and connect everything with inexpensive single-mode fiber from the core router across the data center to all the way to each individual server inside racks. Then never touch it again instead of, as with multi-mode fiber, change the infrastructure every few years.

No kidding, single-mode

fiber is less expensive than

dental floss, yet it regularly

carries the entire internet

(28)

What Data Centers are 10 km or 6.25 Miles Long?

While most data centers are not 2km or 10km long, the “km” spec is another way of stating the optical power of the laser. Measured in powers of ten called dBs (Decibels), the Mellanox PSM4 offers ~3.3 dBs of optical power (LR4 has an optical power of 6dBs) which is enough to push through hundreds of meters of a lossy fiber infrastructure consisting of dirty and/or misaligned optical connectors, jumpers, optical patch panels and other interferences to the light path. This is like needing a very powerful flash light to shine through a dense forest of twigs, branches and leaves in the way even though the actual distance is relatively short.

Multiplexing

Hundreds of different colored laser pulses can be simultaneously sent over a strand of single-mode fiber. Telecom does 1,000 but in the data centers today, four is common and soon headed for 8 wavelengths. Additionally, single-mode fiber is data rates agnostic, unlike multi-mode fiber. Unlike multi-mode fiber, the same fiber can be used for 1G, 10G, 25G, 50G or 100G individual line rates with little impact on the reach. Sending multiple 100G data streams with different colored laser down a single strand of low-cost single-mode fiber is the technology goal of the future.

Single-mode Transceivers in Modern Data Centers

The main single-mode transceiver types used in data centers today are: LR: 10G, 25G, SFP, 1-channel, 2-fiber, LC connector, 10km, 1310nm

PSM4 40G, 100G QSFP, 4-channel, 8-fiber, MPO/APC connector, 500m, 1310nm CWDM4 40G, 100G QSFP, 4-channel, 2-fiber, LC connector, 2km, 1310nm LR4 40G, 100G QSFP, 4-channel, 2-fiber, LC connector, 10km, 1310nm

PSM4 sends each channel into separate parallel fibers –one per channel. CWDM4 and LR4 multiplex 4-channels each with different wavelength lasers into a single fiber and de-multiplexes them at the receiver end – essentially sending a “rainbow” down the fiber. PSM4’s are generally used at less than 500m, CWDM4 up to 2km and LR4 to 10km.

(29)

| page 29

<< Back to TOC

Numerous PSM4 Applications and Configurations for Any Need

The PSM4 has many different configuration application uses and is one of the hottest selling transceivers in the hyperscale segment. It can bus 100G point-to-point over 2km or can be broken out using passive fiber splitter cables or half-AOC hybrid (called a, “pigtail”) into dual 50G or quad 25G links for linking to servers, storage and other subsystems within a rack. The PSM4.MSA specifies 500m reach.

PSM4 Breakouts to Servers & Storage—Beside long reach 500m point-to-point links, PSM4 channels can also be split out individually. The diagram below shows a 100G PSM4 transceiver split using a passive breakout splitter cable with an MPO on one end and either dual MPOs (50G) or quad LC connectors (25G) on the other ends. CWDM4s and LR4s cannot do this feature and can only bus 100Gb/s point-to-point.

(30)

The following diagram illustrates the Mellanox, “end-to-end” system solutions consisting of switches and network adapters with cables and transceivers showing all the different uses of single-mode optics in modern data centers from core routers as far away as 10km all the way to individual servers using various single-mode transceivers and fiber combinations.

(31)

| page 31

<< Back to TOC

Mellanox offers:

• Complete line of single-mode transceivers PSM4, CWDM4, LR4, LR

• Covering all reaches from 100m to 10km and each reach segment is optimized for the lowest cost • Every transceiver is tested in actual switching systems – also designed and built by Mellanox. This

ensures out-of-the-box, trouble free installation and optimum performance.

• All parts are tested to 1E-15 – well beyond IEEE requirements of BER 1E-12 and offering ~1,000 fewer bit errors than competitive offerings.

Key single-mode transceiver

features:

• Enables long reaches up to

10km making various reaches

a non-issue

• Employs inexpensive

single-mode fiber

• Line rate agnostic supporting

100G per channel futures

• Enables multiplexing multiple

(32)

A Glance into the Future

There is an old saying in New England, “If you don’t like the weather, just wait a minute.” This saying is starting to apply to the high-speed interconnect space as well. Cloud computing, HPCs and now converged Cloud/HPCs are driving the link speeds improvements faster than at any other time in history. What used to take 5-8 years is now done in 2-3. Traditional enterprise data centers are just now moving from 1G/10G/40G to 25/50/100G while Cloud and HPC are well into deploying 25/50/100Gb/s and soon 200G and 400G. Now, that’s life in the fast lane!

Newly Announced 200G HDR InfiniBand AOCs

Mellanox announced a new line of LinkX DAC and AOC cables capable of running at a whopping 200G in a QSFP56, 4-channel x 50G configuration!

These cables support the newly announced 200G InfiniBand HDR200 and 100G HDR100 speed Quantum switches and ConnectX-6 host bus adapter cards (network adapters). For the first time in InfiniBand’s history, we have a double-ended, 1:2 cable splitter cable that enables a single 200G HDR port in a switch to be split into two HDR100 ports in host bus adapter cards in servers, storage and other subsystems at 100Gs. Using HDR-Dual HDR100 splitter cables, a 40-port Quantum InfiniBand switch can support 80 100Gb/s HDR100 links. The next speed hop will be called NDR for 400G.

Big Changes for Ethernet & InfiniBand Going Forward

Going faster and faster encounters increasing complexities in electronics and optical physics. So, the transceiver industry is introducing several changes. Sending 2 data bits per clock, increasing the number of channels, and changing the transceiver packaging, and optical connectors. These changes combined with future line rate advances of 100G PAM4 will enable 400G and 800G per transceiver in the future. 1. New Signaling Scheme: For the last 35+ years, digital signaling has been based digital ones or zeros

with one bit per defined clock pulse (called NRZ). Now, the industry is moving to 2-bits per clock pulse by varying the amplitude of the pulse to four levels of 00,01,10,11 (instead of just 1,0) called PAM4 for Pulse Amplitude Modulation 4-channels. The entire infrastructure of switches, network adapters cables and transceivers must change to adopt this new technology. 50G PAM4 allows keeping the same low-cost 25GHz electrical infrastructure but transferring data at twice the rate with 2-bits per clock pulse or 2x25G=50G.

2. Move to 8-Channels: Increasing the number of channel from four to eight provides more aggregate bandwidth but makes everything larger and requires more electrical paths and thermal dissipation. 3. New Transceivers Packaging: QSFP28

supports 3.5-4.5W with four-channels whereas the new packages offer 8-channels and 12-15W support. In July, 2017, the SFP-DD MSA was announced to advance the SFP design from one channel to two channels and from 2W to 3.5W enabling up to 100G and 200G in an SFP-DD form-factor using 50G or 100G PAM4 signaling. Several industry groups are battling for leadership over the next transceiv-er package type. Cisco is leading the QSFP-DD charge and Arista is leading

the OSPF group. Each offer backward compatibility, cooling and connector different options and more cryptic buzzwords than ever before!

(33)

| page 33

<< Back to TOC

Summary

Mellanox is one of the few companies in the business that designs silicon ICs and complete systems for network switches and network adapters as well as ICs for transceivers. This gives Mellanox a unique view into the complicating issues in optimizing total system performance.

In the middle of the diagram below are four of our most popular 25G and 100G Spectrum 32-bit 25G NRZ-based switches.

New is the SN2010 switch which offers eighteen 25G SFP28 ports and four 100G QSFP28 ports. This switch is designed for Ethernet storage as it is becoming clear, especially with NVMe FLASH over Fabrics (NVMeoF), that a network switch is different from a storage switch and plays to the new trend of storage-optimized, memory-centric computing. The SN2100 and SN2010 are half-wide switches and enable mixing and matching in a singe 1RU bay. Every Ethernet rack and row application for network switching can be addressed with these four products. A similar product line is available for InfiniBand switching.

The bottom side middle of this figure illustrates the various Mellanox ConnectX network adapters for 10G-25G line rates and 40G/100GbE quad channel adapters in both single and dual port configurations. Adapters are most often used to uplink servers and HDD, SSD and NVMe storage subsystems to Top-of-Rack switches. See the ConnectX sections of the Mellanox website for many more products and options.

On the left side of the adapters are shown DAC cables in the server/storage rack or what we call “DAC-in-the-Rack”. Virtually every combination of interconnects can be seen in this diagram from 10G/ 25G and 100G single DAC cables to dual-50G and quad-25G splitter cables that enable linking at any line speed to system servers and storage subsystems. All Mellanox switches, network adapters and transceivers are backwards compatible with 10G, 14G, and 25G line rates supporting quad channel versions of 40G, 56G and 100G.

Shadowing DAC cabling on the right side are AOC cables – also in 25G/40G/100G single AOCs as well as dual-50G and quad-25G splitter cables. While AOCs used in the rack are more expensive than DAC cabling, many system builders need to link subsystems residing is multiple racks to a common Top-of-Rack switch and spanning at reaches greater than 3-5 meters maximum of DAC cabling. Subsystems and storage bays are often as far as 10-30 meters away from the server complex along the row and adjacent rows. AOCs and splitters are a perfect, low-cost solution for these applications and less expensive than optical transceivers.

While not displayed in the diagram, SR and SR4 multi-mode transceivers and LR and PSM4 single-mode transceivers can also emulate the AOC splitters using dual MPO and 8-LC fiber splitter cables for applications that need to disconnect the fiber from the transceivers using optical connectors. Again, at a higher cost than AOCs but with the added benefit of being able to disconnect the fibers from the transceivers to rout in infrastructures such as structured cabling.

On the top are aqua-colored, 25G/100G multi-mode transceivers and AOCs. AOCs most often used for 10-30 meter reaches and connectorized SR and SR4 transceivers for structured cabling and reaches up to 100m. To the left are single-mode transceivers in 25G/100G in different types for reaches spanning 500m, 2km and 10km. The longer the reach in optics, the more complex and expensive the transceivers become so there are multiple design types (PSM4, CWDM4, LR4) to optimize for the lowest cost.

(34)

(35)

| page 35

(36)

(37)

| page 37

<< Back to TOC

The modern data center has focused on DAC and AOC cables with multi-mode and single-mode transceivers in 10G and 25G line rates, in single SFP and quad channel QSFP form-factors. Large and hyperscale data center builders are driving down prices and up the rate of change to unprecedented levels. The next phase of the industry starting in 2018 will bring many changes in line rate, modulation techniques, number of channels and new packaging schemes. The jump from 10G/40G to 25G/100G was fairly smooth as it had a minimum number of changes. However, the jump to 50G/200G and 100/200/400G entails very significant changes on nearly every aspect of switching, network adapters, DAC, AOC cables and optical transceivers and optical connectors.

Why Mellanox Cables and Transceivers?

Mellanox cables and transceivers are used in leading industry hyperscale, HPC, enterprise storage and telecom data centers. Mellanox is the only supplier that designs its own switch and network adapter CMOS silicon as well as systems and cards and including designing and manufacturing its own DAC and AOC cables and multi-mode transceivers – including transceiver BiMOS ICs.

• Wide portfolio of leading data center oriented cables and transceivers products form 10/40G, 25/50/100G and 200G in Ethernet & InfiniBand

• End-to-end solutions: switches, network adapters, cables and transceivers for Ethernet and InfiniBand – all designed to work optimally together.

• Mellanox-Designed: Mellanox designs and manufacturer’s its own DAC and AOC cables and multi-mode transceivers – including transceiver ICs. This ensure high-signal integrity and consistent product quality.

• Mellanox-designed transceiver and AOC ICs offers the lowest power consumption in the industry <0.5W 25G SR and 2.2W for 100G SR4 & AOC.

• System-level testing: All products are tested in live switching systems found in data centers, not simulated on a benchtop as with many competitors

• Every product is tested – not batch sample tested or tested on a bench top

• Testing to BER 1E-15 – compared to competitors testing at IEEE standard of 1E-12. 1E-15 is about ~1,000 fewer data errors ensuring optimal performance and data integrity

• Mellanox is a founding member and leader in many of the next generation industry standards at IEEE & IBTA and form-factor MSAs such as:

- SFP-DD, QSFP-DD, OSFP - CWDM8, COBO

Contact your Mellanox sales representative for availability and pricing options for any of the 200 different cables and transceivers products on the Mellanox.com LinkX website. Simple to use and only 3-4 clicks, start to finish, and you are at the product brief PDF downloads.

www.mellanox.com/products/interconnect/

Also, there is link to buy online and have it delivered to your door. Visit the new Mellanox site at:

(38)

Learn more at www.mellanox.com

©NVIDIA, the NVIDIA logo, Mellanox, LinkX, ConnectX, Dynamix QSA, Mellanox Spectrum and Mellanox Quantum are trademarks and/ or registered trademarks of Mellanox Technologies Ltd. and/or NVIDIA Corporation in the U.S, and in other countries. Other company and product names may be trademarks of the respective companies with which they are associated. EBK-LINKX-201802-001

NEW DEVELOPMENTS IN DATA CENTER CABLES & TRANSCEIVERS

The latest wave in rapid data center

growth is resulting from an increase

in machine-to-machine traffic due

to expansion in server virtualization,

software defined networks (SDNs)

and in cloud computing which is

driving massive demand for

high-capacity data center infrastructures.

High-speed interconnects are playing

an increasingly more important role

in these technologies

25G IS THE NEW 10G; 50G THE NEW 40G;

AND 100G IS AMAZING!

NEW DEVELOPMENTS IN DATA

Contents

Key Takeaways

For more information about NVIDIA

Mellanox

LinkX

cables and transceivers.

Introduction

Why So Many Types of High-speed Interconnects?

DAC – A Cabling Industry Shake Up

What is DAC Cabling?

Low-Price, Low-Power DAC Advantages Stimulate its Popularity

Backwards Compatibility

Mellanox LinkX Offers 18 Different DAC Options!

QSFP-to-SFP Adapter Enables Many Options

BER – Designed for High-Performance Computing (HPC)

“Just Use FEC to Clean it up!”

More Value – Extending the Reach Past 3 Meters

Zero Latency Delays

Learn from DAC Disasters

“Some things

in life you don’t

go cheap on –

eye surgeons,

parachutes and

Mellanox DAC Manufacturing

At Mellanox, so called, “PLUG & PLAY” means “PLUG-IN AND WALK-AWAY”;

not the usual Plug & Play-All-Day needed to get things to work.

Every Cable Tested in Real Systems

Key DAC Cable features:

• Lowest cost, high-speed

interconnect solution

• Acts as a plug & play cable

• Available in multiple

configurations for every

application from 0.5 to 7

meters

• Point-to-point links to connect

racks together and breakouts

cables link switches to servers

and storage systems

• QSA adapter enables linking

single channel devices to quad

channel ports in switches and

network adapters

DAC Cables - Why Choose Mellanox?

AOCs - Active Optical Cables

What is an AOC?

What are AOC Features and Advantages?

Key AOC Cable features:

• Lowest cost optical solution

• Acts like a plug and play

cable but offers optical

long reach features without

optical connector hassles

• Available in point-to-point

configurations to link racks

together and breakout

links individual servers and

storage subsystems

• Lower power consumption

& lower maintenance than

optical transceivers

How AOCs Differ from Other Interconnect Solutions

Achieving Lower Product Cost

AOCs Offer Lower Operational “Hidden Costs”

_Mellanox

_LinkX

_{cables and transceivers.}