Photonic Networks for Data Centres and High Performance Computing

Photonic Networks for Data Centres and High Performance Computing

Philip Watts

Department of Electronic Engineering, UCL

Yury Audzevich, Nick Barrow-Williams,

Robert Mullins, Simon Moore, Andrew Moore

Computer Laboratory, University of Cambridge

Motivation

• Future digital services are reliant on power efficient computers

– Data centres, large servers, scientific computing

• Power management is now the primary issue

– 200 W/chip thermal limit reached – Interconnect accounts for majority of

power consumption

• Transistors scale, wires don’t

• Can’t continue to exploit Moore’s Law without dealing with interconnect

power issue

• Many developments are addressing this issue:

– 3D stacked memory

– Non-volatile ‘unified’ memory – Photonic interconnect

130nm CMOS (2002)

45nm CMOS (2008)

Increase

Transfer 32b across chip

20 computations

57 computations

+285%

Transfer 32b off-chip

260 computations

1300 computations

+500%

Power of Computation vs. Communication

MEMORY CACHE CORES PHOTONIC NETWORK

3D Computer Chip

Optical PCB Polymer Waveguides

Optical Backplane

Components of a Photonic C2C Interconnect

Tx

MUX DEMUX

Tx

Chip #1 Chip #2

Tx

Si/SiO₂ Waveguides

400 x 200 nm, Bend Radius ≈ few μm

Polymer waveguides in standard FR4 PCB

(Cambridge) Hybrid Si Laser (UCSB/Intel)

Silicon ring resonators can modulate, filter and switch

Germanium Photodiodes integrated with Silicon waveguides 3 x 1 μm (Intel)

16 x 16 integrated SOA switch (Cambridge/Eindhoven)

X

Characteristics of Photonic Networks

• Photonics offers very high bandwidth (WDM) over long distances with lower power

• Photonics favours circuit switching

– No viable optical buffer exists – End-to-end paths required

• Computer architectures favours packet switching

– Message sizes in Shared Memory systems ≥ 64-bits

• Design implications of complete computer systems with photonic interconnect are not clear

FIFO EO

FIFO OE

FIFO EO

FIFO OE

Optical Network

Edge Buffered Photonic Network

4 3D Mesh

Fat Tree

Traditional Electronic Networks

On-chip Lasers or Off-chip Power Supply

• On chip laser approach

‒ Avoid losses in distribution and getting light onto chip

‒ Can be rapidly power gated (≈ns)

• Photonic power supply (PPS) approach

‒ Only compact and low drive power components need to be on-chip

‒ Single optical power supply can be used for multiple devices

• We can’t store photons – any

generated light is dissipated mainly on- chip

• Unlikely that efficiency of optical power supply will be >20%

‒ Efficiency of electronic power supply can be >80% and is naturally dynamic

LASER LASER

LASER

MOD MOD

MOD PHOTONIC

POWER SUPPLY

On-chip laser approach

Photonic Power Supply approach

5

Photonic Networks for Data Centres - Vision

• Scalability of the all-optical network is limited to around 64 ports, rack

network

‒ Optical power and attenuation

‒ Latency and complexity of

Arbitration/Contention resolution

• Expandable using OEO sections

Optical Network FIFO EO

FIFO OE OE

EO

FIFO EO FIFO OE OE

EO

FIFO EO

FIFO OE

FIFO EO

FIFO OE

Optical Network

FIFO EO

FIFO OE

FIFO EO

FIFO OE

Optical Network

Methodology

Multi- Wavelength

Source

Network Adapter

Receiver Power Request

(optional) Allocator

Processor Chip Switch Chip

• Modelled the power consumption of the photonic path using the photonic power supply approach

• Created SystemVerilog models of the network control circuits and synthesized using a low leakage 45 nm CMOS process

• Evaluated the power consumption of circuit switched and time slotted networks covering a single rack

– How much power is disipated on-chip?

– What can be power gated?

Networks: Circuit Switching

• Electronic Network is

required for residual flows

• Analysis of the traffic

patterns generated running the PARSEC benchmark suite on simulated 32-core x86 system running Linux

• Observe the effect of

changing the circuit decision time

– Circuit decisions are ideal (a priori knowledge of traffic)

Energy per bit – Circuit Switching

10^-7 10^-6 10^-5 10^-4 10^-3 10^-2 10^-1 10⁰ 10^-12

10^-11 10^-10 10^-9

Circuit Decision Time (s)

Energy (J/bit) Current Injection - Min

Current Injection - Max Capacitive - Min

Capacitive - Max

Electronic Packet Switch

Setting up circuits on μs

timescales required

Only 1.0 – 3.2 packets

on average per circuit

No significant advantage in

static case Ring resonator Clos switch assumed

Time Slotted Networks

Scheduled Speculative

Intra-Rack Network Latency

Time slots are 10 ns, sized for transmission of a single 32B cache line

Speculative does not allow

power gating of PPS

Time Slotted Network

10⁵ 10⁶ 10⁷ 10⁸

10^-12 10^-11 10^-10 10^-9 10^-8

Packet Arrival Rate, pkts/s/port

Total Energy, J/b

SOA Switch

Ring Resonator Switch SOA switch, Dynamic OPS) Electronic total energy

10⁵ 10⁶ 10⁷ 10⁸

10^-12 10^-11 10^-10 10^-9 10^-8

Packet Arrival Rate, pkts/s/port

Processor Chip Energy, J/b ^{SOA Switch}

Ring Resonator Switch SOA switch, Dynamic OPS) Electronic total energy

11

Time slotted networks can be > order of magnitude lower power than electronic mesh network

Sources of Power - Time Slotted Network

Total Network Power, including PPS (W)

Sources of Power – Time Slotted Network

Power Dissipation on the Processor Chip, including PPS

Summary

• Photonic time slotted networks can reduce power consumption by > order of magnitude

– Total network power

– Processor chip dissipation

• Further work required to understand how these networks scale under realistic data centre traffic patterns

• Transceiver power consumption is significant proportion of on-chip disipation

– Yury will tell you more in next talk