SC14 Remote I/O Pipeline Processing Demonstrtion

OpenFabrics

Software

User Group

Workshop

SC14 Remote I/O Pipeline

Processing Demonstrtion

#OFSUserGroup

Work done under contract to Naval Research Laboratory

Dardo D Kleiner / CIPS, Corp. /

[email protected]

Linden Mercer / PSU /

[email protected]

Demonstration/Test Motivation

JTRS, Wireless, Mobile, etc …

100Gb/s

10Gb/s

1Gb/s

100Mb/s

1Mb/s

Large Numbers of Lite-Clients

Simple Queries Trigger Extensive Bandwidth Utilization

L

A

T

E

N

C

Y

R

E

D

U

C

T

IO

N

Sensors

Super

computers

Distributed

Processing

Distributed

Processing

People

People

Data

Data

Remote I/O Pipeline Processing

Network

100G

Crypto

SCInet

Oakland, CA

NERSC

Chicago

StarLight

New Orleans

SC14

ESNet

100G

Testbed

Controller

Pipelines

Manager

So ware

Defin

e

d

Exchange

Display &

User Interac on

100G

Crypto

X

X

CTL

CTL

NERSC-TB1

STAR-TB1

AOFA-CR5

Source

Data

24 x 4K

Video

(WTN)

Source

Data

24 x 4K

Video

Pipe-lines

Proc-essing

Ce

nt

ur

yL

ink

10

0G

CenturyLin

k

100G

>95% BW

Efficiency

Gigapixels/sec

Network on

demand

100G Type 1 (ESS)

Risk Reduction

Prototype

Processing on

demand

Network/

crypto on

demand

Processing on

demand

Pipe-lines

Proc-essing

Pipe-lines

Proc-essing

Washington, DC

NRL

90 40

10GE

100GE

Testbed

100GE

Testbed

40GE

4x10GE

4x40GE,

12x10GE

2x100GE

100GE

12x10GE,

4x40GE

AOFA-CR5

WASH-CR5

ANA-200G

Europe

ESnet

100G Testbed

(Starlight)

STAR-CR5

CHIC-CR5

MANLAN

ATLA-CR5

SC14 Scinet,

New Orleans

MAX Switch

ESnet

100G Testbed

(NERSC)

NASH-CR5

NERSC-TB1

HOUS-CR5

KANS-CR5

DENV-CR5

NASA

Goddard

NRL demo (100G)

NASA demo

–

Northern path (25G)

NASA demo

–

Southern path (75G)

75 25 25

Caltech demo

–

ANA-200G CERN (100G)

Brian Tierney, ESnet 11/6/2014

SC14 demos

–

ESnet

CERN

100GE

FNAL

40 40

Caltech demo

–

NERSC path (100G)

100 100

Starlight

Fabric

AMS

100GE

100

VLANs 1802,1810

VLANs 1940-1949

VLANs 2605

VLAN 1860

VLAN 1862

40

VLANs 2602,2603

exoGENI Rack

(Starlight)

Circuit SC04 Circuit SC09,10 Circuit SC01

ELPA-CR5

75

BNL/FNAL demo (40G)

VLAN 2100

University of

Chicago

STAR-TB1

FILENAME

SC14-DEMOS-V11.VSD

BNL

SUNN-CR5

PNWG-CR5

10GE

Singapore

CenturyLink

100GE

CenturyLink

100GE

Univ of Chicago demo (40G)

100 40

exoGENI Rack

(OSF)

40GE

25

All Links

100G unless

otherwise

noted

exoGENI demo (40G)

VLANs 1921,1922

100

Ciena/Canarie

Research Net

100GE

ANL demo (100G)

VLANs 1925

NEWY-CR5

CenturyLink 100GE to SC14 25 CenturyLink 100GE to SC14 SC03 40

NRL

2x10GE

LLNL demo (80G)

Layer 3

Other Demos 10G or Less:

5-7G between PNNL, NERSC, ORNL, ANL, and BNL; DOE Data Replication demo (L3)

10G Loop from U. Chicago to ELPA for UDP/UDT accelerator NIC demo (VLAN: 1911/12)

10G from Singapore to SC14, 5G from Singapore to GA Tech (via ATLA) (VLAN 4000-4004)

10G loop from Aspera Booth at SC14 to Chicago, (VLAN 1930)

90

U Tokyo demo (90G)

VLANs 2987-89

LLNL-CR5

80 Circuit SC02 20 20

100G Network Paths

SC14 Pipelines Processing

Detail

Oakland-NERSC or

SC14-NOLA

Chicago or

Wash., DC (NRL)

SC14 – New Orleans, LA

Switch

X RMA targ.

sink _src sinkOp onalsrc 40 GbE R0CE Adapter RMA targ. sink _src Op onal sink _src RMA targ. sink _src sinkOp onalsrc RMA targ. sink _src sinkOp onal_src RMA targ. sink _src sinkOp onalsrc RMA targ. sink src Op onal sink _src sink sink sink sink src sink sink RMA init. sink src 40 GbE R0CE Adapter Switch X RMA targ.

sink _src sinkOp onalsrc 40 GbE

R0CE Adapter

RMA targ. sink _src sinkOp onal_src RMA targ. sink _src sinkOp onal_src RMA targ. sink src

Op onal sink _src RMA targ. sink _src sinkOp onalsrc RMA targ. sink _src Op onal sink _src sink sink sink sink src sink sink RMA init. sink _src 40 GbE R0CE Adapter Switch X RMA targ.

sink _src sinkOp onalsrc 40 GbE R0CE Adapter RMA targ. sink _src Op onal sink _src RMA targ. sink _src sinkOp onal_src RMA targ. sink _src sinkOp onal_src RMA targ. sink _src sinkOp onalsrc RMA targ. sink src Op onal sink _src sink sink sink sink src sink sink RMA init. sink _src 40 GbE R0CE Adapter Switch X RMA targ.

sink _src sinkOp onal_src 40 GbE

R0CE Adapter

RMA targ. sink _src sinkOp onal_src RMA targ. sink src Op onal sink _src RMA targ. sink src Op onal sink _src RMA targ. sink _src sinkOp onalsrc RMA targ. sink _src Op onal sink _src sink sink sink sink src sink sink RMA init. sink _src 40 GbE R0CE Adapter async i/o

sink _src sinkOp onal_src Adaptec controller async i/o sink src Op onal sink _src async i/o sink _src sinkOp onal_src

async i/o sink _src sinkOp onalsrc

async i/o sink src

Op onal sink _src async i/o sink _src sinkOp onalsrc

RMA init. sink _src RMA init. sink _src RMA init. sink _src RMA init. sink src RMA init. sink _src RMA init. sink src 40 GbE R0CE Adapter async i/o sink _src sinkOp onal_src Adaptec

controller async i/o sink _src sinkOp onal_src

async i/o sink src Op onal sink _src async i/o sink src Op onal sink _src async i/o sink _src sinkOp onalsrc

async i/o sink src Op onal sink _src RMA init. sink _src RMA init. sink _src RMA init. sink _src RMA init. sink _src RMA init. sink src RMA init. sink _src 40 GbE R0CE Adapter async i/o sink src Op onal sink _src Adaptec controller async i/o sink _src sinkOp onalsrc

async i/o sink _src

Op onal sink _src async i/o sink _src sinkOp onal_src

async i/o sink src

Op onal sink _src async i/o sink _src sinkOp onal_src

RMA init. sink _src RMA init. sink src RMA init. sink _src RMA init. sink _src RMA init. sink _src RMA init. sink _src 40 GbE R0CE Adapter async i/o sink _src sinkOp onalsrc Adaptec controller async i/o sink src Op onal sink _src async i/o sink _src sinkOp onalsrc

async i/o sink _src sinkOp onalsrc

async i/o sink _src sinkOp onal_src

async i/o sink src Op onal sink _src RMA init. sink src RMA init. sink _src RMA init. sink src RMA init. sink src RMA init. sink _src RMA init. sink _src 40 GbE R0CE Adapter Switch X 40 GbE R0CE Adapter RMA targ. sink _src Op onal sink _src RMA targ. sink _src Op onal sink src RMA targ. sink _src Op onal sink src RMA targ. sink src Op onal sink _src sink sink sink src sink 4K pres. sink _src NVidia

100G

100G

RMA targ. sink _src Op onal sink _src 40 GbE R0CE Adapter RMA targ. sink _src sinkOp onalsrc RMA targ. sink _src sinkOp onal_src RMA targ. sink _src sinkOp onal_src RMA targ. sink src

Op onal sink _src RMA targ. sink src sinkOp onalsrc

RMA Init. sink _src RMA Init. sink src RMA Init. sink _src RMA Init. sink _src RMA Init. sink _src RMA Init. sink src 40 GbE R0CE Adapter RMA targ. sink _src sinkOp onalsrc 40 GbE R0CE Adapter RMA targ. sink _src Op onal sink _src RMA targ. sink _src sinkOp onalsrc RMA targ. sink _src sinkOp onal_src RMA targ.

i/o sink _src sinkOp onalsrc RMA targ. sink src Op onal sink _src RMA Init. sink src RMA Init. sink _src RMA Init. sink src RMA Init. sink _src RMA Init. sink src RMA Init. sink _src 40 GbE R0CE Adapter RMA targ. sink src Op onal sink _src 40 GbE R0CE Adapter RMA targ. sink _src sinkOp onalsrc RMA targ. sink src Op onal sink _src RMA targ. i/o sink src Op onal sink _src RMA targ. sink _src sinkOp onalsrc

async i/o sink _src sinkOp onal_src

RMA Init. sink _src RMA Init. sink src RMA Init. sink _src RMA Init. sink _src RMA Init. sink src RMA Init. sink _src 40 GbE R0CE Adapter RMA targ. sink _src sinkOp onalsrc 40 GbE R0CE Adapter RMA targ. sink src Op onal sink _src RMA targ. sink _src sinkOp onalsrc RMA targ.

i/o sink _src sinkOp onalsrc RMA targ. sink _src sinkOp onal_src RMA targ. sink _src sinkOp onal_src

RMA Init. sink src RMA Init. sink _src RMA Init. sink src RMA Init. ini ator sink src RMA Init. sink _src RMA Init. sink _src 40 GbE R0CE Adapter

Message Bus

Enterprise

Cloud

Sensor

Platform

Ground

Station

Streams

Sink

Transform

Source

Sink

Transform

Source

Sink

Transform

Source

Message Bus

SC14 Pipelines

Processing Detail

Cont’d

async i/o

sink

src

Op onal

sink

src

Adaptec

controller

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

40 GbE

R0CE

Adapter

async i/o

sink

src

Op onal

sink

src

Adaptec

controller

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

40 GbE

R0CE

Adapter

async i/o

sink

src

Op onal

sink

src

Adaptec

controller

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

40 GbE

R0CE

Adapter

async i/o

sink

src

Op onal

sink

src

Adaptec

controller

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

async i/o

sink

src

Op onal

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

RMA init.

sink

src

40 GbE

R0CE

Adapter

100G

Message Bus

Oakland-NERSC and SC14-NOLA

•

EchoStreams FlacheSAN system

– 48 SSD

– 4 x 40 Gbps

•

Sourcing 24 coordinated

uncompressed UHD video

streams

– 4096x1714x24 fps each

– > 4 gigapixels/second (RGB)

– >97 gigabits/second

•

>150 Gbps raw storage to network

performance

•

RDMA networking provides low

demand on CPU – more

processing available for the

mission

SC14 Demo Quicklook Results

disk2null

disk2roce

disk2tcp

null2roce

null2tcp

SC14 Demo Quicklook Results

From 10//2014

disk2roce

0

GB/

s

ec

10

5

0

100

50

Gb

/s

ec

0

50

%

100%

CPU

Band

w

idth

SC14 Demo Quicklook Results

From 11/7/2014

0

100

50

Gb

/s

ec

Band

w

idth

• Early test between NERSC (Oakland, CA) and StarLight (Chicago)

• Single FlacheSAN2 data source

– >23 Gbyte/second raw storage performance

– Network data output of >150Gbps over four 56Gbps InfiniBand or 40Gbps

Ethernet (RDMA).

– 2RU density

– Minimal power consumption of 450W.

• Data streams are “caught” by 4 compact Pipeline processing

systems, each with 40G Ethernet interfaces

100 Gbit Ethernet Type 1

Prototype

EAI Design Type 1 Prototype

• Conforms to NSA Ethernet Security Specification (ESS)

• Available as EE100G Commercial Off the Shelf (COTS) Encryptor

– 100Gb/s AES-256 XPN-MAC-SEC GCM Encryption Core [IEEE

802.1AEbw]

– Verified in Silicon at 400Mhz in Xilinx Virtex-6 HX565T, Virtex-7 2000t

– 512 Security Associations by default (32-64K configurable)

– 128 Internal TCAM Classifier Rules by default (32-16K configurable)

– Highly Configurable Soft/Technology Independent VHDL to support 100GbE

applications in FPGAs or ASICs.

Results

• 4K Video Processing

NRL and Collaborators Conduct 100 Gigabit/Second Remote I/O Demonstration

Participants and Contributors

Naval Research

Laboratory

Orange

Silicon Valley

DREN

#OFSUserGroup

OpenFabrics Software

User Group Workshop

Thank You

NRL

Booth

1639

CenturyLink

Century

-Link

SCinet

ESnet

NRL SC14 Demo

(all VLANs

are “tagged” unless noted)

0

1

0

1

1

1

1

1940,44,45,49

1941,44,46,49

1943,44,48,49

1940,44,45,49

1941,44,46,49

1942,44,47,49

1943,44,48,49

1

2

3

4

NERSC-ECHO

0

1

0

1

1

1

1

1

STAR-ASUS1

0

1

0

1

1

1

1

1

STAR-ASUS2

1942,44,47,49

0

1

0

1

1

1

1

1

STAR-ASUS3

0

1

0

1

1

1

1

1

STAR-ASUS4

NERSC-TB1

STAR-TB1

AOFA-CR5

LAC-OCC-MLXe

0

1

0

1

1

1

1

1940,44,45,49

1941,44,46,49

1943,44,48,49

1940,44,45,49

1941,44,46,49

1942,44,47,49

1943,44,48,49

1

2

3

4

NOLA-ECHO

0

1

0

1

1

1

1

1

NOLA-STEEL1

0

1

0

1

1

1

1

1

NOLA-STEEL2

1942,44,47,49

0

1

0

1

1

1

1

1

NOLA-STEEL3

0

1

0

1

1

1

1

1

NOLA-STEEL4

NOLA-MLNX

1940,44,45,49

1941,44,46,49

1942,44,47,49

1943,44,48,49

1944,49

0

1

0

1

1

1

1

NOLA-TWR

NRL-MLNX

1940,44,45,49

1941,44,46,49

1943,44,48,49

0

1

0

1

1

1

1

1

NRL-SUPER1

0

1

0

1

1

1

1

1

NRL-SUPER2

1942,44,47,49

0

1

0

1

1

1

1

1

NRL-SUPER3

0

1

0

1

1

1

1

1

NRL-SUPER4

NRL-VDX

1940,44,45,49

1941,44,46,49

1942,44,47,49

1943,44,48,49

core-sw

0

1

0

1

1

1

CT

NOLA-CRYPTO

0

1

0

1

1

PT

NRL-CRYPTO

1945-49

1945-49

0

1

0

1

1

1

PT

0

1

0

1

1

CT

Z

Untagged

Z

Untagged

1940-44,Z

1940-49

1940-49

WASH-CR5

1

9

4

0

,4

1

,4

4

,

4

5

,4

6

,4

9

1

9

4

2

,4

3

,4

4

,

4

7

,4

8

,4

9

MAX

10 GbE

40 GbE

100 G

1639-A

1639-F

5/1/

1

1/1/

1

?/?/

?

6/1/1

3/1/

1

1/1/

1

2/1/

1

A – 1940

B – 1941

C – 1942

D – 1943

E – 1944

U – 1945

V – 1946

W – 1947

X – 1948

Y - 1949

11/10/14 11:08 AM