ENOS: a Network Opera/ng
System for ESnet Testbed
Eric Pouyoul ([email protected])
Technology Exchange
Cleveland, Ohio, September 2015
Is ESnet really developing Yet Another Network
Opera:ng System (YANOS) ?
Focus on:
• Security
• Resource sharing (mul/-‐
applica/ons)
• design paNerns
• ODL, ONOS, other were not
available at the /me.
• ESnet traffic is not typical.
• Not a controller (does not
implement one)
• Will leverage any WAN,
produc/on quality controller
Well, yes, sorry, but
we had to !
Co-‐design with hardware
and network
ENOS Architecture
10/6/15 3
Netshell
security, core services, API, SSHD
Controller Drivers
ODL, ONOS,…
VM Driver
Linux containers (libvirt)
Database Immediately or eventually consistent Stats, Tests perfSONAR, SNMP, Sampling Layer 2 services MulEPoint L2 VPN Layer 3 SDX
REST API, Portal
Network Services Drivers
Netshell Design
Java Virtual Machine
OSGi
Karaf
Netshell Bundles, Python
ENOS Modules
ENOS and SoKware Technologies
10/6/15 5
• Java Virtual Machine (JVM) for a produc/on-‐ready environment
• ENOS security leverages Java Security Manager
• Performance
• Python for rapid so^ware development
• Useful for experiments, quick prototyping
• “The natural language of ENOS”
• ENOS integrates Jython, a Python 2.7 interpreter running in the JVM.
• OSGi, Karaf for module, applica/on management.
>>> from net.es.netshell.api import TopologyProvider >>> graph =
topo.getGraph(TopologyProvider.WeightType.TrafficEngineering) >>> lbl = topo.getNode('[email protected]')
>>> amst= topo.getNode('[email protected]')
>>> from org.jgrapht.alg import DijkstraShortestPath
>>> path = DijkstraShortestPath.findPathBetween(graph, lbl, amst) >>> for link in path:
... node = topo.getNodeByLink(link.getId()
... print "Node= " + node.getId() + "\tlinkId= " + link.getId()
Node= urn:ogf:network:es.net:sunn-cr5 linkId= urn:ogf:network:es.net:sunn-cr5:to_lbl-mr2_ip-a:0 Node= urn:ogf:network:es.net:sacr-cr5 linkId= urn:ogf:network:es.net:sacr-cr5:to_sunn-cr5_ip-a:0 Node= urn:ogf:network:es.net:denv-cr5 linkId= urn:ogf:network:es.net:denv-cr5:to_sacr-cr5_ip-a:0 Node= urn:ogf:network:es.net:kans-cr5 linkId= urn:ogf:network:es.net:kans-cr5:to_denv-cr5_ip-a:0 Node= urn:ogf:network:es.net:chic-cr5 linkId= urn:ogf:network:es.net:chic-cr5:to_kans-cr5_ip-a:0 Node= urn:ogf:network:es.net:wash-cr5 linkId= urn:ogf:network:es.net:wash-cr5:to_chic-cr5_ip-a:0 Node= urn:ogf:network:es.net:aofa-cr5 linkId= urn:ogf:network:es.net:aofa-cr5:to_wash-cr5_ip-a:0 Node= urn:ogf:network:es.net:lond-cr5 linkId= urn:ogf:network:es.net:lond-cr5:to_aofa-cr5_ip-a:0 Node= urn:ogf:network:es.net:amst-cr5 linkId= urn:ogf:network:es.net:amst-cr5:to_lond-cr5_ip-a:0
Drivers
10/6/15 7
Netshell
security, core services, API
• Implements / uses Netshell Generic API: Topology, Monitoring,
• Provides an interface to services that are not implemented in ENOS,
such as the OpenFlow controller
• Some core func/onali/es (DB, messaging) are provided by third part
so^ware requiring a driver.
Driver Applica/on
ENOS Mul:point VPN Service Func:ons
ENOS Path Computa/on ENOS Topology Service OSCARS Driver Open Daylight Driver Virtualiza/on / Isola/on MAC Learning Layer 2 Broadcast DHCP Layer 2 Provisioning Virtual Machine ManagementALBQ AMST ANL AOFA ATLA BNL BOIS BOST CERN CHIC DENV ELPA FNAL HOUS KANS LANL LBL LLNL LOND NASH NERSC NEWY ORNL PNNL PNWG SACR SAND SLAC STAR SUNN WASH ESnet PE Router (2+)x10GE (n)x10GE Testbed Host
Planned SDN Testbed node locations Planned SDN Testbed connectivity overlay (using OSCARS circuits)
ESnet SDN
Testbed
AMST CERN AOFA WASH STAR ATLA DENV LBLESnet SDN Testbed Hardware: Corsa
Technology
OpenFlow 1.3 / 1.4 / 1.5+ Open vSwitch +
• FPGA-‐based SDN switch, with OpenFlow controller interface
• Highly scalable both in number of flows and speed
• Doing common ac/ons (defined by pipeline) very fast
• No broadcast, subset of OpenFlow matches and ac/ons
DP6440 4 x 100G CFP2 Ports 24x 10G SFP+ Ports DP6420 48 x 10G SFP+ Ports 4 x 40G QSFP+ Ports DP6430 2 x 100G CFP2 Ports 24x 10G SFP+ Ports DP6410 24 x 10G SFP+ Ports
100G
10G
ESnet PE Router
(2+)x10GE
(n)x10GE
Testbed Host
ESnet SDN Testbed
Node Logical View
ESnet SDN Testbed
Node Physical View
ESnet PE Router
(2+)x10GE
(n)x10GE
Services VM
• So^ware switch and Services
VM paired with every hardware switch.
• Most flows only pass through
hardware switch (green)
• Flows requiring special handling
go through so^ware switch
(blue), possibly to Services VM
(red).
• Hardware switch provides
performance, stability
• So^ware switch and service VM
provide flexibility, without compromising reliability
ESnet SDN Testbed
Node Logical View
Design PaUerns at work:
Traffic Mirroring.
10/6/15 13 ANL LBL CERN STAR SDN POP LBL SDN POP CERN SDN POP Shared OSCARS layer 2circuits
Private OSCARS layer 2 circuit
Private OSCARS layer 2 circuit Private OSCARS layer2 circuit OVS OVS OVS Flow Flow VM
ENOS Roadmap
Year 1 (2013-‐14)
• Ini/ally only a proof of concept
• Focus Future ScienceDMZ
Year 2 (2014-‐15)
• LBL funding (LDRD)
• Working prototype
• Mul/ple point VPN for high performance flows
Year 3 (2015-‐2016)
• Deploy semi-‐produc/on mul/point VPN with traffic engineering.
• Support research projects: security, intent based networking,…
Lessons Learned and Summary
• Java + Python + SSHD is very powerful and yet simple combina/on.
• Aggrega/ng, grooming and normalizing data such as topology simplifies greatly network aware applica/ons.
• OSGI/Karaf allows us to painlessly change ODL version, ONOS, other OSGi based controller/applica/ons.
• Security is not trivial, especially when using third party so^ware: we needed to turn off security in order to use ODL.
• Will be open sourced very soon.
A network operaEng system does really look like a computer operaEng system. Perhaps Linux itself could be the execuEon environment for network policies.
10/6/15 15
