Gigabit Ethernet Motherboard Tutorial

(1)

2nd eVLBI Workshop 15-16 May 2003, Dwingeloo

High Data Rate Transmission for vlbiGRID

using the Academic Network

Related projects include:

MB-NG, DataGrid, DataTAG, UKLIGHT

Richard Hughes-Jones, Ralph Spencer, Steve Parsley

The University of Manchester & JIVE

(2)

Evaluating Motherboards & Gigabit NICs

u PCI Activity

u Logic Analyzer with

n PCI Probe cards in sending PC n Gigabit Ethernet Fiber Probe Card n PCI Probe cards in receiving PC

Gigabit

Ethernet

Probe

CPU

mem

chipset

NIC

CPU

mem

NIC

chipset

Logic Analyser

Display

u Latency

u Round trip times with UDP frames

n Slope gives sum of individual data transfers end-to-end

n Histograms

u UDP Throughput – Capacity & Available

n Send a controlled stream of UDP frames spaced at regular intervals.

(3)

UDPmon: The Works

K

How it works:

n The ‘Zero_stats’ request also provides an interlock against concurrent tests.

Zero stats _{OK done}

●●●

Get remote statistics

Send statistics Send data

frames at regular

intervals _●●●

Time to send Time to

receive Inter-packet time

Signal end of test

OK done

n bytes

Number of packets

Wait time time

(4)

SuperMicro 370DLE: Latency: SysKonnect

n PCI:32 bit 33 MHz

n Latency small 62 µs & well behaved n Latency Slope 0.0286 µs/byte

n Expect: 0.0232 µs/byte

PCI 0.00758 GigE 0.008 PCI 0.00758

n PCI:64 bit 66 MHz

n Latency small 56 µs & well behaved n Latency Slope 0.0231 µs/byte

n Expect: 0.0118 µs/byte

PCI 0.00188 GigE 0.008 PCI 0.00188

n Possible extra data moves ?

n Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset n CPU: PIII 800 MHz n RedHat 7.1 Kernel 2.4.14 S ys Ko n n e c t 6 4 b it 6 6 MHz y = 0.0231x + 56.088 y = 0.0142x + 81.975 0 20 40 60 80 100 120 140 0 500 1000 1500 2000 2500 3000 Me s s a g e le n g th b yte s La te nc y us S ys Ko n n e c t 3 2 b it 3 3 MHz y = 0.0286x + 61.79 y = 0.0188x + 89.756 0 20 40 60 80 100 120 140 160 0 500 1000 1500 2000 2500 3000 Me s s a g e le n g th b yte s La te nc y us

(5)

SuperMicro 370DLE: Throughput: SysKonnect

n Max throughput 584Mbit/s

n No packet loss >18 us spacing

n Max throughput 720 Mbit/s

n No packet loss >17 us spacing

n Packet loss during BW drop

n Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset n CPU: PIII 800 MHz

n RedHat 7.1 Kernel 2.4.14 SysKonnect 32bit 33 MHz

0 200 400 600 800 1000 0 5 10 15 20 25 30 35 40

Transmit Time per frame us

R e c v W ir e r a te M b it s/ s 50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes SysKonnnect 64bit 66MHz 0 200 400 600 800 1000 0 5 10 15 20 25 30 35 40

R e c v Wi re ra te Mb it s /s 50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes

(6)

SuperMicro 370DLE: PCI: SysKonnect

n Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset n CPU: PIII 800 MHz PCI:64 bit 66 MHz

n RedHat 7.1 Kernel 2.4.14

n 1400 bytes sent n Wait 100 us

n ~8 us for send or receive

n Stack & Application overhead ~ 10 us / node

Receive transfer NIC _{à mem} Send PCI

Packet on the Ethernet fibre 11.6 µs Send setup

Send transfer Mem _{à NIC}

Receive PCI

(7)

SuperMicro 370DLE: Throughput: Alteon

n Packet loss < 10 us spacing

n Max throughput 930 Mbit/s

n Packet loss < 10 us spacing

n Packet loss during BW drop

n Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset n CPU: PIII 800 MHz

n RedHat 7.1 Kernel 2.4.14 Alteon 32bit 33 MHz

0 200 400 600 800 1000 0 5 10 15 20 25 30 35 40 Transmit Time per frame us

R e cv W ir e r a te Mb its /s 50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes Alteon 64 bit 66 MHz 0 200 400 600 800 1000 0 5 10 15 20 25 30 35 40

R ec v W ire r ate M bi ts/ s 50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes

(8)

IBM das: Throughput: Intel Pro/1000

n Max throughput 930Mbit/s n No packet loss > 12 us n Clean behaviour

n 1400 bytes sent n 11 us spacing n Signals clean

n ~9.3us on send PCI bus n PCI bus ~82% occupancy

n ~ 5.9 us on PCI for data recv.

UDP Intel pro1000 IBMdas 64bit 33 MHz

0 100 200 300 400 500 600 700 800 900 1000 0 5 10 15 20 25 30 35 40

Transmit Time per frame us

Recv W ir e ra te M bi ts/ s 50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes

n Motherboard: IBM das Chipset:: ServerWorks CNB20LE n CPU: Dual IBM 340 1GHz PCI:64 bit 33 MHz

(9)

The SuperMicro P4DP6 Motherboard

u Dual Xeon Prestonia (2cpu/die)

u 400 MHx Front side bus

u Intel® E7500 Chipset

u 6 PCI-X slots

u 4 independent PCI buses

u Can select:

n 64 bit 66 MHz PCI

n 100 MHz PCI-X

n 133 MHz PCI-X

u 2 100 Mbit Ethernet

u Adaptec AIC-7899W dual channel SCSI

u UDMA/100 bus master/EIDE channels

n data transfer rates of 100 MB/sec burst

(10)

SuperMicro P4DP6: Throughput Intel Pro/1000

n No packet loss

n CPU utilisation on the receiving PC was ~ 25 % for packets > than 1000 bytes

n 30- 40 % for smaller packets

n Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) n CPU: Dual Xeon Prestonia 2.2 GHz PCI, 64 bit, 66 MHz

n RedHat 7.2 Kernel 2.4.19

gig6-7 Intel pci 66 MHz 27nov02

0 200 400 600 800 1000 0 5 10 15 20 25 30 35 40

R ecv W ire r at e Mb its /s 50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes

gig6-7 Intel pci 66 MHz 27nov02

0 20 40 60 80 100 0 5 10 15 20 25 30 35 40

% C P U I dl e Rec ei ve r 50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes

(11)

SuperMicro P4DP6: PCI Intel Pro/1000

n 1400 bytes sent n Wait 12 us

n ~5.14us on send PCI bus n PCI bus ~68% occupancy n ~ 3 us on PCI for data recv

n CSR access inserts PCI STOPs n NIC takes ~ 1 us/CSR

n CPU faster than the NIC !

n Similar effect with the SysKonnect NIC

n Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) n CPU: Dual Xeon Prestonia 2.2 GHz PCI, 64 bit, 66 MHz

(12)

SuperMicro P4DP8-G2: Throughput Intel onboard

n Max throughput 995Mbit/s n No packet loss

n 20% CPU utilisation receiver packets > 1000 bytes

n 30% CPU utilisation smaller packets

n Motherboard: SuperMicro P4DP8-G2 Chipset: Intel E7500 (Plumas) n CPU: Dual Xeon Prestonia 2.4 GHz PCI-X:64 bit

n RedHat 7.3 Kernel 2.4.19 _{Intel-onboard}

0 200 400 600 800 1000 0 5 10 15 20 25 30 35 40

R e cv W ire ra te M b its/ s 50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes Intel-onboard 0 20 40 60 80 100 0 5 10 15 20 25 30 35 40

% CP U I dl e R ec ei ve r 50 b ytes 100 b ytes 200 b ytes 400 b ytes 600 b ytes 800 b ytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes

(13)

u

All NICs & motherboards were stable – 1000s GBytes of transfers

u

Alteon could handle 930 Mbit/s on 64bit/66MHz

u SysKonnect

gave 720-876 Mbit/s improving to

876-990 Mbit/s on later

boards

u Intel

gave 910 – 950 Mbit/s and

950-995 Mbit/s on later boards

u

PCI and GigEthernet signals show 800 MHz CPU can drive large

packets at line speed.

u

More CPU power is required for receiving – loss due to IP discards

n

Rule of thumb at least 1 GHz CPU power free for 1 Gbit

u

Times for DMA transfers scale with PCI bus speed but CSR access is

constant

n

New PCI-X and on-board controllers are better

u

Buses: 64 bit 66MHz PCI or faster PCI-X are required for performance

n

32 bit 33 MHz PCI bus is REALLY busy !!

n

64bit 33 MHz are > 80% used

(14)

Raid0 Performance

u

Maxdor 3.5 Series

DiamondMax PLus 9 120

Gb ATA/133

u

3ware 7500-8 controller

u

Write

Slight increase with

number of disks

u

Read

u

3 Disks OK

u

Write 100 MBytes/s

u

Read 130 MBytes/s

Read Throughput and no. Disks Stripe 64k raid0

0 20 40 60 80 100 120 140 160 0 200 400 600 800 1000 1200 1400 1600 1800 2000

File size MBytes

Mb yt es /s 2 disk 3 disk 4 disk Write Throughput and no. Disks Stripe 64k raid0

0 20 40 60 80 100 120 140 160 180 0 200 400 600 800 1000 1200 1400 1600 1800 2000

File size MBytes

Mb yt es /s 2 disk 3 disk 4 disk

(15)

Gridftp Throughput + Web100

u

Long term Wire rate:

~520Mbit/s:

u

See alternate 600/800 Mbit

and zero

u

Cwnd smooth

u

No dup Ack / send stall /

(16)

http data transfers HighSpeed TCP

u

Apachie web server

out of the box!

u

prototype client - curl http library

u

1Mbyte TCP buffers

u

Same HW & 2Gbyte file

u

Throughput ~725 Mbits/s

u

Cwnd - some variation

u

No dup Ack / send stall /

(17)

Radio Astronomy VLBI Demos at

iGrid2002 and ER2002

u

Web based demonstration sending VLBI data

n

A controlled stream of UDP/IP packets

n

500 Mbit/s on the production network Manchester

_{à Amsterdam}

32 bit words bits vertical

Bit Stream ~ 62 UDP Packets / Data Track Frame

Timestamp _Data

(18)

European Topology: NRNs, Geant, Sites

SuperJANET4

Brussels

ER 2002

Manchester

Jodrell

SURFnet

Amsterdam

Dwingeloo

(19)

The Works:

n bytes

Wait time time

•••

Raid0

Disc

UDP Data

Raid0

Disc

RingBuffer RingBuffer

TCP Control

(20)

Receiving Packets

Time

1

2

3

4

6

Incoming packet ids

1

2

3

4 Move 6

Continue with 7

u

Packets have

Header:

packet id + timestamp

Data

n Put header directly into control area n Put data directly into ring buffer n No extra copy

u

Assume UDP packet arrive in order none lost

n Deposit data into next consecutive slot

u

Inspect Header

n Move data onward to correct location n Record:

l The received inter-packet spacing

(21)

TCP in Action

Application TCP Network TCP Application

1 ACK 1 ACK 2 Resend 3 2 ACK 2 ACK 2 ACK 2 3 4 5 Permanent Delay Spacing doubled

(22)

The VLBI Demonstration

u

BE & LBE Qos on GÉANT Backbone

(23)

The GÉANT Weathermap

u

Normal Traffic

u

Normal Traffic

+

u

Radio Astronomy Data

n 500 Mbit/s

u

Normal Traffic

+

u

Radio Astronomy Data

+

u

Less Than Best Effort

n 2.0 Gbit/s

(24)

2nd eVLBI Workshop 15-16 May 2003, Dwingeloo To NNW From NNW

Traffic Mbit/s

Manchester Campus Access links:

iGrid2002

u

Manchester – SuperJANET

Access link

n 1 Gbit/s

u

500 Mbit UDP/IP

u

70% Usage

u

24-26 September 2002

(25)

Manchester Campus Access links:

ER2002

u

By 10 November 2002:

u

Tests Prior to the Demo

u

Manchester – SuperJANET

Access link

n 1 Gbit/s

u

500 Mbit UDP/IP

u

Incoming depressed

u

UKERNA offered us a

private 1 Gbit direct

connection

(26)

GÉANT Backbone during ER2002

u

Dante Backbone

u

UK-NL link

n Incoming at NL

u

500 Mbit/s VLBI data

(27)

Traffic QoS Classes on GÉANT Backbone

u

Normal Traffic

u

Normal Traffic

+

u

Less Than Best Effort

n 2.0 Gbit/s

u

Normal Traffic

+

u

Radio Astronomy Data

n 500 Mbit/s

u

Normal Traffic

+

u

Radio Astronomy Data

+

u

Less Than Best Effort

n 2.0 Gbit/s

(28)

Some Measurements made during ER2002

No LBE 0 2 4 6 8 10 12 14 16 18 20 0 20 40 60 80 100 120 140 160 180 200 Transfer number No . O ut of or de r 0 5000 10000 15000 20000 25000 No . L os t num_badorder num_lost

With 1.8Gbit LBE

0 5000 10000 15000 20000 25000 30000 35000 40000 45000 0 20 40 60 80 100 120 140 160 Transfer number No . O ut of o rd er 0 5000 10000 15000 20000 25000 No . Los t num_badorder num_lost

u

1.24 M packets

0.4 %

(29)

1-way delay during iGrid2002

VLBI Data Transfer Times

5135000 5140000 5145000 5150000 5155000 5160000 5165000 0 200 400 600 800 1000 Packet Number Pa ck et R e ce iv e t im e us 7800 8000 8200 8400 8600 8800 9000 9200 9400 9600 1 W ay La te nc y us recv_time one_way time us

(30)

Latency Histograms

256 bytes UDP manw-sara0_8nov02

0 5 10 15 20 25 14500 14550 14600 14650 14700 14750 14800 14850 14900 Latency us N( t)

0 5 10 15 20 25 30 14500 14550 14600 14650 14700 14750 14800 14850 14900 Latency us N( t) C

0 10 20 30 40 50 14900 14950 15000 15050 15100 15150 15200 15250 15300 Latency us N( t)

No LBE

2.0 Gbit LBE UK

_{à DE}

0 10 20 30 40 50 14900 14950 15000 15050 15100 15150 15200 15250 15300 Latency us N( t)

0 20 40 60 80 100 120 15150 15200 15250 15300 15350 15400 15450 15500 15550 Latency us N( t)

0 20 40 60 80 100 15150 15200 15250 15300 15350 15400 15450 15500 15550 Latency us N( t)

(31)

Throughput

manw-sara0_7nov02 0 100 200 300 400 500 600 700 800 900 1000 0 5 10 15 20 25 30 35 40

Delay between sending frames us

Recv W ir e r at e M b it s/ s 50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes manw-sara0_LBE2.0G_7nov02 0 100 200 300 400 500 600 700 800 900 1000 0 5 10 15 20 25 30 35 40

Recv W ir e r at e M b it s/ s 50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes manw-sara0_8nov02 0 100 200 300 400 500 600 700 0 5 10 15 20 25 30 35 40

out of o rd er 50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes

No LBE

2.0 Gbit LBE UK

_{à DE}

manw-sara0_LBE2.0G_7nov02 0 500 1000 1500 2000 2500 3000 3500 0 5 10 15 20 25 30 35 40

out of or de r 50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes Scale Change

(32)

Packet re-ordering

u

20 µs Transmit packet spacing 10,000 sent

u

Different behaviour for small packets

u

Still some re-ordering at 40 µs spacing

n ~100 for 100 bytes n 20 for >1000 bytes 0 100 200 300 400 500 600 0 0.5 1 1.5 2 2.5 LBE rate Gbit/s

nu m ou t o f or de r 1472 1200 1000 600 200 100 50

(33)

Lambda to Dwingeloo

u

Became Operational during iGrid2002

u

Data from Manchester were sent by ftp to Dwingeloo (Westerbork data)

u

Received on PCEVN with Gigabit Ethernet

(34)

Tests on the MB-NG Development Network

u

Sending from Manchester to London

u

Reading form 3ware Raid0

u

Full 1500 byte Ethernet packet transmitted every 12.48 µs on Gigabit Ethernet

u

Rise to 8 µs indicates interaction between reading data from disk and buffering it

in the IP stack

0 100 200 300 400 500 600 700 800 900 0 5 10 15 20 25 Packet spacing us Th roupu t M b it /s Data rate Wire rate

(35)

UKLIGHT

NETHER

-LIGHT

CERN

StarLight

UKLIGHT Global Optical Research &

Optical Switching - Testbed

NorthernLIGHT

CANARIE

CESnet

TERAGRID

Translight

UltraLight

(36)

NETHER

-LIGHT

UKLIGHT@ULCC

“service hub”

StarLight

CESnet

TERAGRID

UKLIGHT in UK

“Development Networks”

u

Proposal includes

u

Exchanging VLBI data

JB - JIVE

u

Bulk HEP data

(37)

K

The production network can transfer VLBI data at over 500 Mbit/s

K

Simple UDP/IP is fine

K

Packet loss acceptable provided link not saturated

K

End Hosts must have sufficient power –

n

“Server Quality” Motherboards

n

High speed disk sub-systems

n

Well designed NIC

K

Many thanks to all who helped

(38)

Acknowledgements

u

Boston Ltd supply of the SuperMicro servers & donation of SCSI disks

n Dev Taygi, Alex Gili-Ross

u

Brunell University

n Peter van Santen

u

Dante

n Marian Garcia, Nicolas Simar, Dale Robertson, Roberto Sabatino,

u

JIVE

n Steve Parsley, Sergei Pogrebenko

u

Metsähovi Radio Observatory

n Ari Mujunen

u

SURFnet & Universiteit van Amsterdam

n Pieter de Boer, Erik.Radius, Cees de Laat, Antony Antony, Mieke van de Berg

u

UKERNA

n Robin Arrack, Bob Day, Jeremy Sharp

u

University of Manchester

n Richard Hughes-Jones, Ralph Spencer, Paul Burgess,

(39)

More Information & Some URLs

K

VLBI Demos at iGrid2002 and ER2002

www.hep.man.ac.uk/~rich/VLBI_web

http://www.hep.man.ac.uk/u/rich/VLBI_web/igrid2002_astro/iGrid2002_v8.pdf

u

UDPmon / TCPmon kit + writeup

http://www.hep.man.ac.uk/~rich/net

u

ATLAS Investigation of the Performance of 100Mbit and Gigabit Ethernet

Components Using Raw Ethernet Frames

http:// www.hep.man.ac.uk/~rich/atlas/atlas_net_note_draft5.pdf

u

DataGrid WP7 Networking:

http://www.gridpp.ac.uk/wp7/index.html

u

DataTAG

http://datatag.web.cern.ch/datatag

u

Motherboard and NIC Tests:

www.hep.man.ac.uk

/~rich/net/nic/GigEth_tests_Boston.ppt

http://datatag.web.cern.ch/datatag/pfldnet2003/papers/hughes-jones.pdf

u

PFLDNet Workshop

(40)

(41)

Throughput Measured for 1472 byte Packets

995 Mbit/s 70-70 µs 990 Mbit/s

0-0 µs SuperMicro P4DP8-G2

Chipset: Intel E7500; PCI 64bit 64 MHz RedHat 7.2 Kernel 2.4.19-SMP 950 Mbit/s 70-70 µs 876 Mbit/s 0-0 µs SuperMicro P4DP6

Chipset: Intel E7500; PCI 64bit 64 MHz RedHat 7.2 Kernel 2.4.19-SMP 930 Mbit/s 400-120 µs 790 Mbit/s 0-0 µs IBM das Chipset: CNB20LE; PCI 64bit 32 MHz RedHat 7.1 Kernel 2.4.14 910 Mbit/s 400-120 µs 720 Mbit/s 0-0 µs 930 Mbit/s SuperMicro 370DLE

Chipset: ServerWorks III LE PCI 64bit 64 MHz RedHat 7.1 Kernel 2.4.14 584 Mbit/s 0-0 µs 674 Mbit/s SuperMicro 370DLE;

Chipset: ServerWorks III LE PCI 32bit 33 MHz RedHat 7.1 Kernel 2.4.14 IntelPro1000 SysKonnect SK-9843 Alteon AceNIC NIC Motherboard