Comparison of Network Interface Controllers for Software Packet Processing (Final Talk)

Comparison of Network Interface Controllers for

Software Packet Processing (Final Talk)

Alexander Frank

Advisors: Paul Emmerich, Sebastian Gallenmüller, Dominik Scholz Supervisor: Prof. Dr.-Ing. Georg Carle

October 2, 2017

Chair of Network Architectures and Services Department of Informatics

Motivation - General

Used software:

• MoonGen - High-speed packet generator based on Lua scripts

• libmoon - Lua wrapper for DPDK

• DPDK (Data Plane Development Kit) - Provides drivers and libraries

for fast packet processing Hardware support:

• DPDK supports NICs from Intel, Mellanox, Broadcom, Chelsio, ...

• MoonGen/libmoon (technically) support all hardware supported by

DPDK

Motivation - Introduction to MoonGen/libmoon

DPDK

NIC NIC

MoonGen

libmoon script MoonGen script

config API config API HW So ft w a re  DPDK libraries  Drivers

 Lua Wrapper for DPDK  Custom Drivers  Rate Control  Timestamping libmoon

Motivation - Goals of the Thesis

Programming: Integrate Mellanox NICs into libmoon/MoonGen

• Change build system to automatically handle Mellanox drivers and

dependencies

• Enable hardware filtering

• Uniform packet counting and statistics

Research: Compare three selected NICs in terms of features and

ca-pabilities

• Hardware level - interfaces, offloads, timestamping

Tested NICs

Vendor NIC driver

Intel Ethernet Controller X550T ixgbe

Intel Ethernet Controller XL710 i40e

Mellanox ConnectX-4 Lx mlx5

Table 1: Investigated Intel and Mellanox NICs

Hardware Interfaces

All three NICs provide similar types of interfaces

Relevant for throughput: network interface and PCIe interface

NIC Series Ports Port Speed PCIe Interface

X550 1/2 10 Gb/s v3.0 x4 @ 31.504 Gb/s

710 1/2/4* 40 Gb/s v3.0 x8 @ 63.016 Gb/s

ConnectX-4 1/2 100 Gb/s v3.0 x16 @ 126.032 Gb/s

Table 2: Investigated Intel and Mellanox NIC series. Port speed refers to usage of a single port [3], [6], [4]

Timestamping

Timestamping is used by MoonGen for high precision latency mea-surements

Drawbacks of software timestamping:

• Inaccurate as software does not know when exactly a packet leaves

the port

Timestamping - Intel

• X550 - 80 MHz clock with precision ± 12.5 ns

• 710 - 625 MHz clock with precision ± 0.8 ns (Values depend on

configured link speed, 40 Gb/s in this case)

0 1 0 1 0 1 0 1 0 1 0 1 1 1 0 0 0 0 0 0 0 bit time Ethernet Start of Frame Delimiter First Octet Following Start of Frame Preamble Octet Message Timestamp Point

Timestamping - Mellanox

Software SQ (Send Work Queue) RQ (Recieve Work Queue) 1. Post WQEs

(Work Queue Entries)

Hardware

WQ

2. Pass WQE ownership to hardware 3. Access/Execute WQEs CQ (completion queue) 4. Post CQE 5. Read CQE

(Completion Queue Entry)

Timestamping - Summary

Point in time to generate timestamp:

• Intel’s NICs timestamp is generated when a certain bit leaves the

port

• Mellanox’ NICs include a timestamp of CQE generation

• When is a CQE exactly generated? Deterministic?

DPDK

DPDK libraries mlx5 PMD Kernel Verbs User Verbs (libibverbs+libmlx5) ConnectX-4/5 mlx5 verbs provider (mlx5_ib) mlx5_core PRM Data Path MoonGen/libmoon User Kernel Hardware Control Path DPDK libraries

I40E PMD/IXGBE Driver

XL710/X550T igb_uio or uio_pci_generic Data Path MoonGen/libmoon User Kernel Hardware Control Path

DPDK with Intel

• DPDK was originally

developed by Intel

• Usage of UIO modules

→ driver completely in userspace

• Registers of hardware

are directly mapped

into userspace DPDK libraries IXGBE Driver X550T igb_uio or uio_pci_generic Data Path MoonGen/libmoon User Kernel Hardware Control Path

DPDK with Mellanox 1

Mellanox uses the InfiniBand software stack. Why? Mellanox Background:

• Portfolio is centered around InfiniBand capable hardware

• Adapter types: VPI (InfiniBand and Ethernet), EN (Ethernet only)

• We use EN types. Software stack is the same for VPI and EN

InfiniBand Background:

• InfiniBand provides very low latencies and is mainly used for HPC

• Verbs are abstract definitions of NIC operations defined in the In-finiBand standard

DPDK with Mellanox 2

• mlx5 PMD is the userspace

driver and interfaces with the verbs layer

• mlx5_core is the kernel level

driver

• Kernel Verbs and the mlx5

verbs provider implement the InfiniBand verbs

• This setup allows to receive

and send packets via the OS’ network stack on ports which are not used by DPDK

Benefits of InfiniBand - RDMA

Buffer Application

Buffer Transport Protocol Driver

Buffer NIC Driver Buffer Sockets

Buffer Application

Buffer Transport Protocol Driver

Buffer NIC Driver Buffer Sockets Buffer NIC Buffer NIC Buffer Application NIC Driver Sockets Buffer Application NIC Driver Sockets Buffer RNIC Buffer RNIC

Traditional Interconnect RDMA Zero-Copy Interconnect

DPDK support

Timestamping:

• The ixgbe and i40e drivers implement timestamping

• The mlx5 driver does currently not support timestamping

• Patchwork DPDK is checking a patch to solve this issue

HW-Filtering:

• Intel devices support DPDK’s old filtering framework

• Mellanox supports this framework only partially

• DPDK 17 introduces new framework with better support by

MoonGen/libmoon Improvements

Task Status

Build process for mlx5 and mlx4 done

Update to DPDK 17.05 done

Integrate DPDK’s generic flow API done

Statistics and uniform packet counting done

Mellanox installation instructions done

Update to DPDK 17.08 done

temporary fix

• A commit in DPDK 17.08 introduced a bug which prevents our

ConnectX-4 Lx from receiving packets

• Temporary workaround is to disable the faulty feature (vectorized

Future Work

Extend support of MoonGen:

• Add support for mlx4 devices (ConnectX-3 Pro) or NICs from other

vendors (e.g. Terminator 5 from Chelsio)

• Faster NICs (> 10 Gb/s)

• Observe development of timestamping capabilities (Mellanox)

[1] InfiniBand FAQ. http://www.mellanox.com/related-docs/whitepapers/InfiniBandFAQ_FQ_100. pdf, 2014. Revision 1.3, accessed on 09/29/2017. [2] Mellanox DPDK. https://community.mellanox.com/docs/DOC-1502, 2014. Updated on 07/26/2017, accessed on 09/29/2017. [3] Intel Ethernet Controller X550 Datasheet.

http://www.intel.com/content/www/us/en/embedded/products/networking/ ethernet-x550-datasheet.html, 2016.

[4] Mellanox Adapters Programmer’s Reference Manual (PRM), Supporting ConnectX-4 and ConnectX-4 Lx.

http://www.mellanox.com/related-docs/user_manuals/Ethernet_Adapters_ Programming_Manual.pdf, 2016.

Revision 0.40, accessed on 09/29/2017. [5] DPDK, Network Interface Controller Drivers.

http://dpdk.org/doc/guides/nics/index.html, 2017. Release 17.11.0, accessed on 09/29/2017.

[6] Intel Ethernet Controller 710 Series Datasheet.

http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/ xl710-10-40-controller-datasheet.pdf, 2017.