Here Now - an Open Source Project Near You

(1)

Here Now - an Open Source Project Near

You

The Linaro LNG Open Data Plane Initiative

Mike Christofferson, Enea

(2)

FOUNDED

1968

TEN OFFICES IN NORTH AMERICA, EUROPE AND ASIA REVENUE

~70 M

USD NO. OF EMPLOYEES

426

 Increasing data traffic in communication devices

require new and innovative software solutions to handle bandwidth, performance, and power requirements, as well as scalable systems management and availability solutions

 A robust product portfolio

 Enea operating systems software is heavily used in wireless Infrastructure (Macro, small cell), gateway, etc. Enea Solutions run in more than 50% of the world’s 8.2M radio base stations.

 Enea provides a commercial Linux

distribution, built by Yocto, with focus on real-time

 Proven, mature middleware solutions for over 10 years – High Availability, Systems

Management, and real-time database

 Global presence, global development, and headquartered in Stockholm, Sweden

(3)

0 20 40 60 80 100 120 140 160 1995 2000 2005 2010 2015 2020 2025 2030

• Super-linear growth in:

– Number of users

– Number of connected devices

– Number of over the top (OTT)

applications and protocols

– Number of (standard) protocols (RFC’s) – Bandwidth usage – Power consumption of network infrastructure

What Is Happening

(4)

• Increasing and varying QoS requirements

– Realtime (e.g. VoIP, video conferencing, gaming)

– Streaming (e.g. music, video)

– Messaging (e.g. IM, Ajax, M2M/IoT)

– Bulk (web data, file sharing, OTA updates)

• More functionality and services implemented in the

network

– Web caching

– Content delivery (CDN)

– Intrusion detection and prevention (IDS/IPS)

– User-specific service level agreements (SLA)

(5)

Consequences

• Need flexibility of software and programmable hardware

– Trend towards software-realised networking - function defined by software

• Need familiar programming environments with robust tools

– For TimeToMarket-driven development of new protocols and services

• Need portability

– Move functionality and applications between hardware platforms optimised for different power/performance/cost points

• Need high abstraction

– To enable innovation in efficient implementations

– E.g. OpenGL/OpenMAX

• Need efficient support for virtualization

– Decouple functionality (SW) from capacity (HW)

– Dynamic partitioning of common HW for different functions

(6)

Solution – HW/OS

Develop and deploy networking applications on general

purpose processors/architectures

• Increasingly ARM and x86

• Users and partners are drawn to the big ecosystems around these architectures

Networking applications running in Linux user space

Develop, debug and deploy using standard Linux tools

• Robust user space access to networking HW resources

Linux enhanced to provide bare metal-like environment

Bare Metal Linux

Avoid TLB misses, interrupts, context switches, system calls, thread migration Direct HW access from user space

Applications run isolated in user space on dedicated cores, unaffected by the Linux kernel and other applications

Optional real-time support

(7)

What Is Open Data Plane?

• ODP is an open source cross-platform framework

for data plane applications

• Common API for application portability

• Multiple implementations tuned to different

platforms for performance

• Result: Easy app portability and performance

Application Environment

• Applications run in Linux user space with

essentially zero system overhead

(8)

Open Data Plane: The Time has Come

• Networking silicon vendors have evolved data plane SDKs for years

– No cross-industry group has sanctioned any common interface on diverse silicon

• The Linaro Networking Working Group - a consortium of 12 networking

stakeholders surveyed the open source landscape

– Consensus: No ideal “one-size fits all”, implementation for diverse hardware/software approaches

• A truly open source & open contribution & cross-platform data plane interface, driven by a cross section of stakeholders, is needed

• Based on the OpenGL model: A software API at a higher level of abstraction, that could offer flexibility of implementations underneath that suit diverse needs.

– The Linaro non-profit open source software engineering organization is launching just such a collaboration…

(9)

Open Data Plane API

• Standardized data plane API to enable Linux-based

networking applications across any architecture

– Open support for ARM, Intel, MIPS & PowerPC !

• Structured to enable future innovation

– Lightweight abstraction preserves performance without prescribing lower –level processing structure

– Access and management of HW accelerators

– Supports optional schedulers to provision easy management and traffic load balancing

• Proprietary SDKs sit underneath for OEM/operator

software platform simplification (e.g.

Supports

DPDK on x86, USDPAA on QorIQ, etc

)

9

Enabling an efficient, truly cross-platform standardized

data plane processing model

Application and

services

portability

across a choice

of hardware

platforms

(10)

ODP Foundational Principles

Event Machine

–

Work-driven many core data plane processing

SoC Abstraction

–

Portable API’s for access to HW/SoC resources

Bare Metal Linux

(a.k.a. Bear Metal Linux)

–

Minimal overhead and deterministic execution in Linux user

space

EM

SoCA BML

(11)

ODP Foundational Principles (2)

• A data plane/networking API and runtime

– Loosely based on the NSN Event Machine

– Event/work-driven and polled programming models

– Portable API’s for accelerators and offloads

– Runs in user space under Bare Metal Linux for best performance and

determinism

• Common API, optimised implementations

– Separately owned and maintained API (e.g. OpenGL)

– Generic portable reference implementation from Linaro

– HW-optimised (possibly proprietary) implementations from

networking SoC vendors

(12)

API and Concepts

• ODP loosely based on Event Machine, originally developed by NSN

– Generic framework for scalable multi-core programming, not limited to packet processing

• Event based abstraction and programming model for handling IO

– Supports packet flows, physical and virtual network interfaces, accelerators, SW endpoints, etc.

– Events represent different types of data: packets, timers, baseband data, HW notifications, SW messages...

• Supports scheduler based programming model (both HW & SW)

– Scheduling of IO events using different algorithms and knowledge of work in progress

– Implicit synchronisation and mutual exclusion between threads • Supports different IO load balancing approaches

– Chose best configuration for traffic profile, latency/throughput requirements, and HW characteristics without changing the application

(13)

EM Basic Concepts

Queue groups Queues with events Scheduler Cores/threads associated with queue groups

Idle core

Queues can be dynamically created and added to and removed from queue groups

Cores can be

dynamically added to and removed from queue groups

Event handlers associated with queues IP-fwd NAT GTP-U DPI RoHC

(14)

Work Scheduling

• Actual scheduling algorithms implementation dependent

• Scheduler can enforce ordering/mutual exclusion

– Parallel, parallel/ordered and atomic queues

– Application doesn’t need software mutex for protecting per-flow state

• Logical flows/queues mapped to hardware queues (if available)

‘Pull’ work, on demand scheduling Clusters/Cores/Threads Logical flows/queues thousands to millions Flows/ QoS classes _WRR SP Scheduling algorithms

WRR - Weighted Round Robin SP - Strict Priority Work Scheduler Processing packet from flow A Processing packet from flow B Idle (power-gated) because of low load

(15)

Dynamic vs. Static Load Balancing

• Networking SoC’s have hardware

suitable for dynamic load balancing

– Queues associated with producer

– Queue and buffer mgmt in hardware

• Server NICs designed for termination

– Static load sharing (based on hashing)

– Queues associated with consumer

– Queue and buffer management in software/shared memory

• Static increases average and worst case

latency and buffer space

– OK for ~8 cores… but not many-core ready (some networking SoC’s already have 30+ cores/HW-threads)

• Static makes core elasticity very

difficult (per-core state with application level seamless/lossless handovers)

– Limits opportunity for power scaling

0 20 40 60 80 100 120 140 1 2 4 8 16 32 64 128 A ver ag e la tency/u S Number of cores Static load-sharing Dynamic load-balancing 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 0% 20% 40% 60% 80% 100% P o w er % Load

Static Load Sharing with typical DVFS Dynamic Load Balancing Power gating + DVFS

(16)

Elasticity and Multi-core Load Balancing

Issues

– Traffic load and pattern varies over time

– Industry trend is to use more cores and more power-efficient cores

– To hit the sweet spot PPA (Power/Performance/Area) – Enabled by inherent parallelism in networking

– Ideally use as many (or few) cores as traffic load and SLA’s require and use them efficiently

ODP

 Supports hardware scheduler and dynamic load balancing

 Cores can be added and removed

 No fixed allocation of cores for specific application

 Enables power or clock gating of idle cores

 Cores can share load dynamically

 Increased throughput,

 Decreased packet latencies,

(17)

Scalable and Elastic Timer Support

Issues

• Many protocols need timers, often several timers per

flow/connection

– Millions of flows in core network means millions of timers

• Timers, like packets, are associated with flows/connections

– Need mutual exclusion of flow context when processing a timer event

ODP



ODP schedules timers together with packets

(18)

Power and Performance Management

Issues

• Traffic load varies

– Daily variation and intermittent bursts

• Use as many or as few cores needed to meet bandwidth and QoS

requirements

– Add and remove worker threads/cores

– Adjust clock frequency of active cores

– Power or clock gate inactive cores

ODP

 Supports power/performance management

 Provides API for observing queue lengths

 Idle worker threads may yield to OS for background tasks or power down core  Application can monitor traffic load and quickly react to increasing load

(19)

vSwitch Integration

Issues

• Efficient and robust integration with software or hardware-accelerated

vSwitch

• No loss of performance for virtualised networking applications using

the dataplane API

ODP



ODP’s queue-based I/O hides actual device implementation

 A queue may represent an actual network interface, a vSwitch port, a

pipeline of further processing stages (e.g. for encryption or encapsulation) etc.

 Allows for HW to copy packets between application and vSwitch

(20)

Openness and Cross-platform

ODP provides:



Support for multiple architectures and platforms (e.g., ARM,

x86, and MIPS)



Open source and an open collaboration

 Not controlled by any single company

 Anyone may join in

 Reference implementations are open source



Based on the Event Machine which currently is implemented

on a number of different HW targets (using

ARM/MIPS/PPC/x86 processors)

(21)

Status Core API Definitions

API Component Description Status

BUFFER Shared memory, buffer pools, buffer types and access functions

Preliminary done, but still work in progress

CLASSIFICATION Ingress packet classification Preliminary work underway, CRYPTO Algorithmic and protocol offload for crypto, hashing, RNG Proposal being implemented

IPC Inter-process communication control plane/data plane TBD PACKET I/O Network interface abstraction Done QUEUE Buffer queue management Done TIMER Protocol timers, periodic ticks Done SCHEDULER Ingress scheduling and distribution to threads/cores Done

Version 0.2. of the API spec available now Version 1.0 by year end 2014

(22)

Status Implementations

Platform Description Status

linux-generic Generic, portable reference implementation, uses Linux facilities (e.g. NetMap, crypto)

Implements

BUFFER/CRYPTO/PACKET-IO/QUEUE/TIMER/SCHEDULE R

linux-dpdk Implementation for x86 using DPDK as the acceleration layer.

Just started

linux-keystone2 HW-accelerated implementation for TI Keystone2 Tracking linux-generic linux-qoriq HW-accelerated implementation for FSL DPAA In progress

(23)

• Cisco will demonstrate real app running on multiple

HW-implementations of ODP

– Usage of API’s

– Usage of HW acceleration through ODP API’s (e.g. ordered and atomic scheduling, crypto)

– Portability

• NSN has had early influence on general architecture and APIs

• Huawei is promoting ODP in public presentations and

expressing their support in meetings

• Ericsson’s new influence in this project

Demos at Linaro Connect USA in Sep’ 14

Contributions and Interest from Major TEMs

The ODP API Specification can be influenced by anyone in the open community