MoonGen. A Scriptable High-Speed Packet Generator Design and Implementation. Paul Emmerich. January 30th, 2016 FOSDEM 2016

(1)

Chair for Network Architectures and Services Technical University of Munich (TUM)

MoonGen

A Scriptable High-Speed Packet Generator Design and Implementation

Paul Emmerich

January 30th, 2016 FOSDEM 2016

Chair for Network Architectures and Services Department of Informatics Technical University of Munich (TUM)

(2)

Source: www.spirent.com

(3)

MoonGen

I MoonGen is a software packet generator

I Cheaper than hardware boxes

I More flexible

I Key features

I Fast: LuaJIT, DPDK1, explicit multi-core support

I _Flexible: _{Craft packets in real-time in user-defined Lua scripts} I Timestamping: Utilize hardware features found on modern

commodity NICs

1_{Intel Dataplane Development Kit, http://www.dpdk.org}

(4)

Multi-threading in Lua

I No native support, some libraries exist

I Our solution: multiple independent LuaJIT VMs

I Maps to our problem domain

I Generate different traffic flows in different threads I _{Inter-thread communication rarely needed}

I Modern NICs support multiple independent queues natively I Serialization (via Serpent2_{) and C functions for (slow)}

inter-VM communication

2_{https://github.com/pkulchenko/serpent}

(5)

Chair for Network Architectures and Services Technical University of Munich (TUM) Architecture MoonGen Core DPDK Userscript Mo onGen HW NIC NIC Port Q0 ... Qn Port Userscript Lua VM Userscript spawn Userscript slave Lua VM Userscript master Lua VM

config API data API

(6)

Example: Generating Load

1 function loadSlave(queue)

2 local mempool = memory.createMemPool(function(buf)

3 buf:getUdpPacket():fill()

4 end)

5 local bufs = mempool:bufArray() 6 while mg.running() do

7 bufs:alloc(60)

8 for i, buf in ipairs(bufs) do

9 local pkt = buf:getUdpPacket() 10 pkt.ip4.src:set(math.random(0, 2^32 - 1)) 11 pkt.udp.src:set(math.random(0, 2^16 - 1)) 12 end 13 bufs:offloadUdpChecksums() 14 queue:send(bufs) 15 end

(7)

Packet data in Lua

I Different protocols are combined to build protocol stacks

I Endless combinations (e.g., tunnels in tunnels...)

I Efficient access to all protocol header fields required

I Dynamically build LuaJIT FFI cdata structs

1 vxlanPkt = createPacket("eth", "ip4", "udp",

2 "vxlan", {"eth", "innerEth"}, {"ip4", "innerIp4"}) 3 local pkt = vxlanPkt(buf)

4 pkt.innerIp4:setSrc("10.0.0.1")

I Dynamically create “class” for the whole protocol stack

I Extremely fast modification operations

I Memory layout defined by cdata struct (can be sent out directly)

(8)

Packet data in Lua

I Different protocols are combined to build protocol stacks

I Endless combinations (e.g., tunnels in tunnels...)

I Efficient access to all protocol header fields required

I Dynamically build LuaJIT FFI cdata structs

1 vxlanPkt = createPacket("eth", "ip4", "udp",

2 "vxlan", {"eth", "innerEth"}, {"ip4", "innerIp4"}) 3 local pkt = vxlanPkt(buf)

4 pkt.innerIp4:setSrc("10.0.0.1")

I Dynamically create “class” for the whole protocol stack

I Extremely fast modification operations

I Memory layout defined by cdata struct (can be sent out directly)

(9)

Summary

I User-defined Lua scripts instead of configuration or DSLs

I LuaJIT is really really fast

I LuaJIT FFI cdata for packet structs

I ≥ 10 Gbit/s per CPU core (≥ 15 million packets per second)

I Execute user-defined script code for each packet

I LuaJIT FFI and C libraries for low-level stuff (drivers)

(10)

Q & A

Try MoonGen yourself!

https://github.com/emmericp/MoonGen

Questions?

(11)

[Backup slide] Performance I: Lua can be faster than C

I UDP packets from varying source IP addresses

1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 5 10 15 CPU frequency [GHz] P ack e t rate [Mpps] MoonGen Pktgen-DPDK

I Pktgen-DPDK needs a complicated main loop that covers all possibilites

I MoonGen can use a tight inner loop

(12)

[Backup slide] Performance II: heavy workload and multi-core scaling

I Generate random UDP packets on 2 10 Gbit NICs

I 8 calls to Lua’s standard math.random per packet

I CPUs artificially clocked down to 1.2 GHz

1 2 3 4 5 6 7 8

0 10 20 30

Number of 1.2 GHz CPU cores

P

ack

et

rate

[Mpps]

(13)

[Backup slide] Performance III: 40 GbE

I Generate random UDP packets on 2 10 Gbit NICs

I 8 calls to Lua’s standard math.random per packet

I CPUs artificially clocked down to 1.2 GHz

64 96 128 160 192 224 256 0 10 20 30 40 50

Packet size [byte]

Rate

[Gbit/s]

1 core 2 cores 3 cores