Contents lists available at ScienceDirect
Computer
Networks
journal homepage: www.elsevier.com/locate/comnet
Review article
A
survey
on
energy
efficiency
in
software
defined
networks
Mehmet
Fatih
Tuysuz
a ,∗,
Zekiye
Kubra
Ankarali
b,
Didem
Gözüpek
c aDepartment of Computer Engineering, Harran University, Sanliurfa, TurkeybDepartment of Computer Engineering, Istanbul ¸S ehir University, Istanbul, Turkey cDepartment of Computer Engineering, Gebze Technical University, Kocaeli, Turkey
a
r
t
i
c
l
e
i
n
f
o
Article history: Received 6 May 2016 Revised 2 December 2016 Accepted 19 December 2016 Available online 21 December 2016 Keywords:
Software-Defined Networking (SDN) Energy efficiency
OpenFlow
a
b
s
t
r
a
c
t
Widedeploymentanddenseusageofcomputernetworksmaycauseexcessiveenergyconsumptiondue totheincreaseinprobabilityofnetworkcongestion,framecollisionsandpacketdroppingratesresulting fromlate-receivedframes.Besides,basedonthedramaticincreaseinnetworkcomplexitywithwireless, mobileandsplittunnelconnections,weakvisibilityintonetworkflowsandhighcostofsomeofpublic andprivatenetworkservices,currentnetworkscanalsobeimpliedasinefficientintermsofboth per-formanceand economy.Software-DefinedNetworking (SDN)isanovelnetworkingarchitecture, which providesadirectlyprogrammableand(logically)centralizednetworkcontrol,separatesnetworkcontrol fromforwarding,andenablesprogrammablenetworkcomponents.SDNcanhaveasignificantrolein re-ducingtheaforementionedexcessiveenergyconsumptioncausedbydatacenters,networkcomponents, and endhosts.In thispaper,1 weexamine theprinciples,benefits, and drawbacksofup-to-date SDN
approachesthatfocusonenergyefficiency.Wealsoprovideabriefcomparisonofpossibleenergygain ratiosofexistingapproaches,discussiononopenissuesandaguidelineforfutureresearch.
© 2016ElsevierB.V.Allrightsreserved.
1. Introduction
In currentcommunication networks, setting up a network re-quires multiple software and hardware-based networking tech-nologies,suchasprotocols,switchesandrouters.Therefore,wired andwireless communicationnetworks, such aspersonal, local or wide area networks, may face with a fundamental complexity problemincasethey consist of a highnumberof connected de-vices having different characteristics andrequirements. Addition-ally, devices in traditional data networks need to be configured individually,taskupdatesofdevicesaretime consuming, andthe evolutionof the hardwarefunctionality iscontrolled onlyby the provider.Therefore, implementinga newnetwork policy may re-quireconfiguringmanydevices.Furthermore,traditionaldata net-works have devices that are designed to perform specific tasks, suchasswitchesandrouters.Hence,theyhaveastaticarchitecture andresultin slow-evolvingnetworkfunctionality. However,there isahighamountof increasein theuseofmobile devices,server virtualization,andcloudservices.Alloftheseactivitiesrequire dy-namicnetworkstructure.
∗ Corresponding author.
E-mail addresses: [email protected] (M.F. Tuysuz), zekiyetekin@ std.sehir.edu.tr (Z.K. Ankarali), [email protected] (D. Gözüpek).
1 This work is supported by the Scientific and Technological Research Council of Turkey (TUBITAK) under grant no. 114E245.
In the past decade, network traffic pattern has also changed. Applicationsnowrequireaccessingdifferentserversanddatabases sinceuserswanttoreachapplications,infrastructureandallother IT resources.Mobile devicesarecurrentlyused toaccessthe cor-porate networks;hence,IT staff becomes obligedto protecttheir data. In addition, data centers are growing. Although intercom-munication between these centers has been evolving, the ever-growing communication demandis mostlyaddressed by increas-ingthebandwidthandsettingupnewnetworkcables.Asaresult, handlingwithmassivedatasetsiscostlywithtraditionalnetworks. Sincenetworkvirtualizationisnowwidelyused,physicalelements cancontainmorethanonevirtualnetworkstructure.However,the reconfigurationprocessisnotagile,easy,andquick.Besides,users’ network functionality is limited; it depends on the vendors and thehardwareproviders.Consequently,all oftheabove-mentioned factsconstitutenovelchallengesfornetworkowners.
As the network size increases,its complexity grows exponen-tially.Inthisregard,currentnetworkinfrastructurerequiresmore flexible and dynamic network operations, programmability and easily modifiednetwork devices.Therefore, thenetwork must be programmedaccordingtoitschangingrequirements.Currently,the key paradigm to achieve this behavior is Software-Defined Net-working(SDN).SDNsimplyprovides a(logically)centralized con-trol point in the network. In an SDN architecture, control plane - the plane that understands the network and decides the flow paths, and data plane - the plane responsible for the transmis-http://dx.doi.org/10.1016/j.comnet.2016.12.012
sion of packets, are separated. Separation of control layer from thedatalayerenablesprogrammability,increasesfunctionality,and providesremotemanagementbetweeninfrastructuresusinga sin-gle openprotocol.Thisstructureallowsnetworkandbusiness ap-plicationstoworktogetherwiththehelpofanalyticsandto recon-figurethenetworkpoliciesaccordingtothechanginguser experi-enceandapplicationperformance.Inthiscontext,networkdesign andarchitecture remainthesamewhileapplicationsandsystems progresstoanadvancedlevel.
InSDN,networkintelligenceandstatearelogicallycentralized, andthe underlying network infrastructureis abstracted fromthe applications [1] . Networkdevices merely need to accept instruc-tions fromtheSDNcontroller insteadofunderstanding all proto-cols.Hence,anynetworkelementcanbechangedinstantaneously. SDN enables the network to meet instantly emerging needs of businessandinstitutionsandhelpthemtocustomizethenetwork. SDN can alsobe beneficial in cases likedirecting devices to safe flows, improving network performance, preparing network band-width for scheduled data transfers, keeping network slices apart with controllerto keep away from research traffic, transition be-tweenmultipledatacentersandsingledatacenter,etc.
In SDN, communication between infrastructure and control layer is provided by the OpenFlow protocol [2] , which was re-leasedbyStanfordUniversityandCaliforniaUniversityin2011and currentlymanagedbytheOpenNetworkingFoundation(ONF). Al-though thetermsSDNandOpenFloware closelyrelated, theyare not interchangeable.WhileSDNisan emergingnetwork architec-ture that enables networksto be programmable witha high de-gree of automation, OpenFlow is a protocol that configures net-work switches in order to provide the communication between the SDN controller and the devices. OpenFlow allows SDN con-trollerstodecidethepathofnetworkpacketsthroughthenetwork ofswitches.However,controllersshouldalsoincludeorchestration tools to dynamically and automatically respond to the needs of thenetwork.Moreover,controllersshouldbeabletocommunicate with other controllers regardless of their own proprietary inter-facesandscriptinglanguages.
Inanutshell,SDNtechnologyaims toaddresstheproblemsof thetraditionalnetworksanditiscurrentlysupported bythe net-workcommunity.Itcanprovidecentralizedcontrolofthenetwork even when thenetwork consistsofmulti-vendor elements.It re-ducesthe complexitythrough networkautomation. Moreover, via SDN,IT staff caneasilyadaptthenetworktotherapidlychanging needs andrequirements. In addition, SDN enables more security sinceitprovidesaglobalcontrolpointoverthenetwork.SDN con-trollercaneasilyregulateanypolicy;therefore,ITstaff cancontrol eventhesmallestelementsofthenetwork,suchasdevices,users, andapplications.
Another inefficiency of the current networking technology is thehighamountofenergyit consumes.Current networksare in-efficient both environmentally and economically (i.e. CO2 emis-sion, operational costs, etc.) and hence they should be reconfig-ured. Inthepast,research scope oftheInformationand Commu-nication Technology (ICT) was mainly basedon performance and cost.Theresearchcommunityputinsufficientefforttotheenergy consumed by ICTs andtheir impact on theenvironment. Current trends,suchasincreasingelectricitycosts,reservelimitations,and increasing emissionsof carbon dioxide(CO2) are shifting the fo-cus ofICTtowards energy-efficient andwell-performedsolutions. Even though governments and companies are now aware of the massive carbonemissions andenergyrequirements, it isobvious thatcarbonemissionsandtheamountofenergyconsumptionwill continueto increase [3] .As statedby theSMART 2020study [4] , ICT-basedCO2 emissionsarerising atarateof6% peryear. With such a growthratio,it isexpected thatCO2 emissions causedby ICTswillreach12%ofworldwideemissionsby2020.
Communica-tion networksdesigned according to thisenergy efficiency crite-riaare calledgreen networks [5] .Inthis context,SDN procedure canhaveasignificantroleinreducingtheenergyconsumptionby decouplingnetworkfunctionalitiesandutilizingsome ofthelocal and network-related parameters. In other words, the global net-work knowledge and centralized decision-making mechanism of SDNmakeitaproperenvironmentforrealizinggreennetworks.
There have been manyreviews focusing on SDN paradigm in theliterature,suchastheworkspresentedin [6–9] .However, ex-tensiveanalysisofSDNintermsofenergyefficiencyhashitherto receivedlittleattention.Towardsclosingthisgap,wefirstpresent the importance and possible ways of energysaving in SDN, and thenexaminetheprinciples,benefitsanddrawbacksofup-to-date SDNapproachesthat takeenergyefficiencyintoaccount.We also presenta brief comparison of possible energy gain ratios of ex-istingapproaches, discussion onopen issues,and a guideline for futureresearch.
Therestofthepaperisorganizedasfollows. Section 2 presents backgroundinformation onthe definition,classification, and pro-cedure of the SDN paradigm. Section 3 presents the importance ofenergysavinginSDN. Section 4 examines theprinciples, bene-fitsanddrawbacksofup-to-dateSDNapproachesintermsof en-ergy efficiency. Section 5 compares theexisting approachesfrom theviewofenergygain,andfinally Section 6 reportstheexisting issues,challengesandpossiblefutureresearchdirections.
2. Background
Although the term SDN has been coined in the last decade, theconceptbehindtheSDNhasbeenevolvingsince1990s,driven by the need to offer user-controlled management of forward-ing in network nodes [10] . In 1990s, researchers proposed pro-grammablenetworksasa solutionforcurrentissues of network-ingandnewideasaboutprogrammable networks, suchas OPEN-SIG,Active Networking,andDCAN, startedtocome out.OPENSIG [11] proposed a method that enables the control of the network hardwarewithaprogrammablenetworkinterface.Active Network-ing [12,13] suggestedaprogrammablenetwork infrastructure, uti-lizinguser-programmableswitchesandcapsules,whichwere pro-gram fragments (carried in user messages) interpreted and exe-cutedbyrouters.DCAN [14] ,whichissimilartoSDN,aimedto de-couplecontrolandmanagementlayers fromnetworkdevicesand insertedthemintootherentities.
In2000s,severalmethods similarto SDNhavealsobeen pro-posed.In thiscontext,4D Project [15] aimed todesigna network withthree layers:a decisionplane, a discovery plane (which in-forms the decision plane about the network to control the data plane)anda controlplane. NETCONF [16] isanother examplefor SDN-likeapproaches.Ithasa layered structureanda mechanism that can install, change or delete the configurations of the net-workdevicesaccordingtotheneedsofthenetwork,andprovidea secureenvironmentwhiletransportingtheconfiguration environ-ment [17] . In addition,Ethane [18] wasthe immature version of OpenFlow.The project managedpolicies andsecurity ofthe net-workusingacentralizedcontroller.
Fig. 1 shows a comparison of the architectures of Traditional and Software-Defined Networks. In traditional networks, control anddata planesare unitedin a singlenetwork device. However, to enable programmability, increase functionality, and enable re-mote management betweeninfrastructures, SDN concentrates on fourkeyfeatures:(i)Decouplingofthecontrolandthedataplane, (ii) Detailed observation of the network with a centralized con-troller,(iii)Openinterfacesbetweendevicesincontrolplane (con-trollers)andthose indataplane, and(iv) Programmabilityofthe networkby external applications.In addition,asshownin Fig. 2 , thereare threelayers in an SDN;(i) Infrastructurelayer (deploys
Fig. 1 . Traditional Networking versus SDN.
Fig. 2 . Simplified view of an SDN architecture.
OpenFlowstandardsandphysicalelements),(ii)Controllayer (pro-videscontrolplaneovertheentirenetwork),andApplicationlayer (transfers instructions or requirements to the controller). Infras-tructurelayer, alsoknown asthedata plane, consistsofnetwork devicesthatsupportaSouthboundAPI,whichallowsdata switch-ingandforwarding.SouthboundAPI,i.e.OpenFlowprotocol,isone ofthestandardized protocolsofONF andprovidesthe communi-cationbetweeninfrastructureandcontrollayersinadditionto
reg-ulatingthecontroller-switchinteraction.NorthboundAPIisthe in-terfacethatprovidescommunicationbetweencontroland applica-tionlayers.Using NorthboundAPI,applicationsinteractwith con-trollersandexplorethenetwork statein ordertomanipulatethe servicesofthenetwork,suchasgoverningthetraffic,path compu-tation, andsecurityoranyother orchestrationoperations [7,8] .In controllayer,SDNcontroller(s)centralizesthecontrolfunctionality ofthenetwork.Controllersareplacedbetweenforwardingdevices
andapplications, andmanage the flow ofdatathrough switches. Finally, the applicationlayer iswhere business andSDN applica-tions,suchasnetworkvisualization,securityapplicationsand rout-ingalgorithms,aredeployed.
3. Why do we need to save energy
Worldwide energyconsumption ofICTequipmentexhibitsthe urgent need for energy efficiencyin networking. In [19] ,authors evaluatetheimpactofdifferentsectorsofICTonenergy consump-tionandCO2 emissions,comparinganearlyreport(GartnerGroup Report,2007 [20] ) aswell asmorerecentreports(EINSEuropean Project,2013 [21,22] ).Authors statethatalthoughtheinitial Gart-ner Reporthasan alarmistperspective,andthepotentially explo-sive growthofenergyconsumptionby ICThasnot been substan-tiated,worldwideICTenergyconsumptionandCO2emissionsstill continuetogrow.Authorsalsomentionthattheimprovingenergy efficiencyofICTequipmentleadstoaslowergrowthofthese met-ricsthanthe increaseoftheworldwideusage ofICT.Inthis con-text,ICTcanalsocontributetoareductioninenergyconsumption andCO2 emissions inother sectors, utilizingefficient networking andcommunicationstrategies.
Despite manydifferencesbetween mobile andfixed networks in terms of their design, implementation, maintenance, monitor-ing, andoperation, thereisatrade-off between energy consump-tionandperformanceinbothnetworks.Thistrade-off,thelevelof which variesin mobile and fixed environments, should be taken into account during system design. For instance, energy-efficient operation maynecessitate thesending ofadditional control mes-sages, such as sleep messages from the SDN controller to the switches, and this additional control traffic decreases the band-widthefficiency.Inmobile wirelessnetworks,energy-efficient op-eration implies reduced power consumption of network devices, whichinturndecreasesthetransmissiondataratesincedatarate andtransmissionpowerarerelatedtoeachotherduetoShannon’s formula. The sleeping of nodes in both wired and wireless net-worksmayincreasethetransmissiondelaynotonlybecauseofthe reduction indataratebutbecauseinsomesituations thepackets needtobequeuedwhilewaitingforthedevice(orotherdevices) towakeup.
Incommunicationnetworks,energyefficiencycanbeachieved via twofundamentalchoices:(i) dataratecan beincreased, orin other words,packetizationdelaywhich istheamountoftime re-quired topush thepacket’s bits intothe wired/wirelessmedium, can be reduced and then terminals can be put into sleep mode for longer time to consume less power within transmissions, or (ii)dataratecanbereducedandhencelowerpowerconsumption canbe achievedattheexpenseoflonger datatransmissiontimes [23] .In contrast,ina networkmanagedby SDN, various parts of theSDNstructurecanbeconfigureddynamicallytoreducepower consumption.Onewayistosettheflowaccordingtothetrafficof thenetworkandputtheunuseddevicesinthenetworkintosleep mode. When there is low traffic load, instead of the whole de-vice,certainportscanbe putintosleepmode.Anothermethodis tooptimize/reduce thememorysizeusedby forwardingswitches asflowtablesare storedinTernary ContentAddressableMemory (TCAM),whichisexpensiveandpower-hungry.However, reducing TCAMtoomuchmaycausefrequentflowentryreplacementswhen newflowsaretobeinstalled.Hence,moreaccessestoTCAMmay consumemoreenergy.Besides,SDNcontrollercanplacenewrules aboutenergyefficiencyintoswitchesundersomeconstraints,such asnumberofswitches androuting policies.Controllercanaimto minimize the energy consumption whilereconfiguring new rules according to thesystem. Virtualizationof servers is another way toreduceenergyconsumption.Thisway,multiplevirtualmachines canworkonthesamephysicalserver.Thus,insteadofusingmany
servers inefficiently,constructing virtual machinescan save addi-tionalenergy [24] .
4. Energy efficient SDN approaches
GreenNetworkinghas beenwidely studiedandnumerous so-lutionshavebeenproposed intheliterature.Mostoftheseworks canalso be adaptedto theSDN concept.Forinstance,authors in [25] presentananalytical modelthatcomparesthe trade-offs be-tweennetworkperformanceandenergysaving.Inordertocreate theiradaptivemodel,theyuseAdaptiveRate(AR)andLowPower Idle(LPI)transmissiontechniquesandperformanceandpower lev-els of Advanced Configuration and Power Interface (ACPI) stan-dard. According to their work, packets are transferred in longer times with the AR. However, LPI also results in delay, as it re-quiresthesystemtosleeporwake-upfrequently.LPIalsoincreases burstiness of the traffic dueto the same frequent sleep/wake-up initiations. In contrast,AR smooths the traffic. Considering these trade-offs,authorscreatetheiranalyticalmodelthatminimizesthe power consumption subject to latency and loss probability con-straints.Theycomparetheir modelwitharouterthat includesAR and LPI capabilities for fixed power and performance levels and showthat their optimizationmodelallows energysavingroughly about 16–17% in comparison to the fixed configuration scenario. As mentioned earlier, these types of green networking solutions canalsobeadaptedtotheSDNandadditionalenergygaincanbe achieved.However,throughoutthissection, we examineonly the worksrelatedtoenergyefficiencyinSDN.
SDN,networkvirtualization(NV),andnetworkfunctions virtu-alization(NFV)are threeterminologiesthat areoftenerroneously usedinterchangeably.SDNhasthreedistinctive features:(i) Sepa-rationof dataplane andcontrol plane, (ii) (Logically)centralized controlunit,(iii)Networkprogrammability.NV,ontheotherhand, createslogicalsegments inan existingnetworkbylogically divid-ingthe networkatthe flow levelirrespective ofthe programma-bilityof thenetwork. It creates a tunnelin the existing network ratherthanphysicallyconnectingtwodomains, andthereby mak-ing the life of network administrators easier by eliminating the burdenofchangingthephysicalnetwork.WhileNVisbeingwidely used for decades, NFV is a relatively new concept referring to putting a service/network function on a tunnel, i.e., virtualizing networkfunctionsandmigratingthemtogenericservers.Thisway, NFV aims to reduce deployment costs for services such as load balancingandfirewallsetup.Themajor differenceofNVandNFV fromSDNisthatNVandNFVaddvirtualtunnelsandfunctionsto thephysicalnetwork,whereasSDNchangesthephysicalnetwork. In other words, SDN is a new concept to provision andmanage the network,while NV andNFV are mechanisms to provide fea-turessuch asresource sharingandnewfunctionswithout having tochangetheinfrastructureofthenetwork.NVandNFVcanexist onexistingnetworks;however,SDNrequirestheconstructionofa newnetworkbecauseitisbasedonacompletelydifferentnetwork managementmechanism.
Withtheenormousincreaseinthe numberofmobiledevices, networkoperatorsneedtoprovidenewservicesandincrease capa-bilitiesinordertomeettheever-growingdemands.Inthisregard, theinnovationspeedhastoincreasewhilethecostsdecrease. Cur-rentnetworkdevicesarehighlyspecializedandyetitisstillhard tocatchthespeedofchangeintheenvironment.Thecostof phys-icalinfrastructures ofmobile networking systemsishighaswell. Sharing of physical resources may reduce this cost. However, it isnotpreferredamongmobile networkoperatorsdueto thelack of regulations. Forinstance, sharing physical resources is forbid-densinceregulatorsthinkitwilldamagethecompetitionbetween operators.Toincrease efficiency(e.g.energy,performance and re-sourceefficiency)anddecreasethecost innetworking,authorsin
[26] proposeasolutiontotheaforementionedissues,by integrat-ingtheSDNandnetworkvirtualization.Althoughnetwork virtual-izationisaneffectivetechnologyinusesincedecades,its integra-tionwithSDN isa novel concept bringing numerousadvantages. In this scope, SDN provides programmable and simpler network devicesthat reduce thecost andenablesmore efficient network-ing.Networkvirtualization providesdifferentoperatorsto use ac-tive sharing,which is concerned withusing the same infrastruc-turebutsplittingit intoslices. Thisway,it willprovide different servicesand make thecompetition among operators possible. To seetheimpact ofthe integrationof SDNandnetwork virtualiza-tion on cost andefficiency, authors compare three different net-workscenarios;(i) classical (where specializednetwork elements arein use), (ii) SDN-based(where SDN-enabled network compo-nentsare in use) and (iii) sharing (where network virtualization andnetwork sharingbetweendifferentoperatorsusingFlowVisor controllers are in use). In thiscontext, authors demonstrate that the total number of networking devices, which is related to the expectedamountoftotalpowerconsumption,willdecreaseinan SDN-basedscenario.Besides,thecostofrepairingandtestingwill alsoreducesincenetworkswillconsistoflowernumberofdevices. Layer-basedarchitectureandtheseparationofcontrolanddata plane in SDNenable dynamicnetwork configuration, which may alsolead to numerousenergy-aware newstrategies. In this con-text, to evaluate existing energy-aware SDN approachesin detail anddiscusstheirstrengths andweaknesses,thissectionisbroken downintosubsections basedontheapproach/metricsusedinthe studies.
4.1. Flow-controlled energy-efficient SDN approaches
Green Abstraction Layer (GAL) proposed in [27] is one ofthe traffic-aware energy-efficient SDN approaches.GAL enables inter-nal communication among network devices to exchange power-relateddata,such aspowerconsumption ofaparticularswitch,a state,oradevice.Thedataisthentransferredtothecontrollerand hence, the controller applies real-time control strategies among networking devices. Energy Aware States (EASs) in GAL are de-finedbylogicalentitiesofthenetwork(i.e.ports,switches,actions ontraffic flow, etc.), fromthepoint ofenergyconsumption, such asenergy consumptionata particularstate or atmaximum sup-porteddatarate.EnergyOptimizermoduleusesinformationabout EASsandcalculates thebeststateforobtaining minimumenergy. GALconsistsofthreephases:discovery,provisioningand monitor-ing.Discovery phase enablesthe controllerto obtain information aboutpowerconsumption of each entity.Provisioning phase sets the state of the network nodes according to the optimal energy stateof thenetwork. After re-arrangingthe networkstate, moni-toringphasesendstheinformationaboutthecurrentstateofeach entity.Finally,LocalControlPolicies(LCPs)optimizethechoice be-tween energy saving and network performance according to the traffic.
Toprovideenergyefficiencyandmanagetheavailableresources in SDN, authors in [28] propose a network management system withthe help ofGreen Abstraction Layer (GAL) [27] . The system takestopology,resources, andtraffic demandasinput and calcu-latesflows andenergystatesof each elementthat will take part inflows. All possiblepaths for agiven flow arecalculated anda listofnodesispreparedtogetherwithnodes’capacities andtheir numberofoccurrences.Toshowtheeffectivenessoftheproposed scheme,authorsimplementanemulationenvironmentand inves-tigate the total network level consumption of a day-night traffic profile; composed by six macro-flows, where 24 hours of a day havebeen divided into 300 time slots. Results show a good ac-curacylevel,asthe trafficdecreases, thelinksenter lower power statesandthetrafficprocessingandforwardingcontinueregularly.
Theworkin [29] alsoaimstooptimizeenergyconsumptionin SDN.TheauthorsfirstprovideaMixedIntegerLinearProgramming (MILP) formulationandthen proposeaStrategicGreedyHeuristic algorithm. In their proposed algorithm, accessnodes of the traf-fic flow informtherest ofthe nodesabouttheir traffic loadand thenthecontroller setsan optimizedtrafficflow andturnsoff as manydevicesaspossible.Throughoutthepaper,authorssetthree constraints; (i)Capacity constraintmakes surethat linksare effi-ciently usedand thereare no underusedoroverloaded links, (ii) Trafficdemandconstraintensuresthatalltrafficdemandsare allo-catedto relatedpathswithenough capacities,and(iii)Node-link relationconstraintshutsdownthenodewhenallofthelinks con-nected to it areturned off.In orderto evaluate the efficiencyof theproposed algorithm, authorsconstitute two networks,a cam-pusnetworkandameshnetwork,usingthreelevelsoftraffic de-mands:low(night),mid(averagedaytime),andhigh(yearlypeak). Theresultsshowthattheproposedmethodcansavehighamount ofenergy(upto45%),especiallyatnighttime.
In order to optimize energy efficiency in SDN, authors in [30] present the GreenSDNapproach, which integratesthree dif-ferent protocols that operate at different layers of the network: AdaptiveLink Rate(ALR) [31] ,whichisachip-level protocol, Syn-chronizedCoalescing (SC) [32] ,which isactive atnode-level,and Sustainability-oriented Network Management System (SustNMS) [33] , which operates at network level. ALR works on links and changes data rates according to the traffic load of the network, whereas SC protocol worksas LPI. However, while LPI works on individualpartsofthenetworkdevices,SCcanputawholedevice into the idle mode. SustNMS controls the network andbalances thetrade-off betweenQoSandenergyefficiencytoquicklyrespond to changing traffic patterns. The work makes use of Mininet as theemulation toolandPOXasthecontroller. Inaddition,sinceit is importantto control theQoS, check the efficiencyof the traf-ficengineering,andcompute theexpectedamountofenergy con-sumption, authors also exploit OpenFlow protocol tools for net-workmonitoring.
The implementation of GreenSDN architecture follows mainly three steps. First, the monitoring part is implemented. This part utilizestheinformationaboutthetrafficbeginningfromthe edge-nodes and continuing through used paths. Second, power states are implemented.GreenSDNdefines twotypes ofpowerprofiles: a loadproportional profileanda linear profile.While theformer consumesenergyproportionalwiththeworkload,thelatterworks witha constant energy.Finally, green energyefficiency protocols are implemented.ALR protocol increasesthe linkrate whenit is turned on. SC compares the total number of packets per second withapreviouslydeterminedbuffer.Ifitishigherthanthebuffer, SCisturnedoff tomeettheworkload;otherwise,packetsareput togethertoprovidemaximumenergysaving.Asmentionedbefore, SustNMSisresponsiblefromthewholenetwork.Ittriestoprovide energyefficientnetworktrafficbytakingintoaccountallthe traf-fic flows,used paths, andalsothe switchesthat will be putinto sleepmode.
In [34] authors propose Exclusive Routing (EXR) to improve fair-sharingrouting(FSR),whichisacommonrouting methodfor fairallocation.FSRselectsasubsetoflinksanduniformlyspreads flows across the links without anydelay. However, this behavior indicatesthatall linksworkwithlessthanfullcapacity. Through-out the paper,authorsdemonstrate that FSRapproach usesmost ofthe switches withutilizationlower than 55% ofits full capac-ity.Inthiscontext, authorsclaimthat efficientuseofalready ac-tivated links, i.e. full use of their capacity,and turning off more switches will decrease energy usage even more. Thus, the main ideaistoeliminatelow utilizationoflinksandswitches.EXRhas two kindsofflows: active and suspended .The controllermanages thewholeprocess,notifiestheswitchesaboutlettingorblockinga
flow.Whenanewflowarrives,ifthereareanyidlepathswithout anyflowonit,theflowissetasactiveanditissent.Ifthereisno idlepathbutthereisapathwithsuspendedflowswithlower pri-oritiesthanthecurrentflow,thentheflowissetasactiveanditis sentviathispreemptivepath.Ifthisisnotthecase,theflowisset assuspended.Ifmorethanone optionisavailable,thepath with theminimumnumberofidleswitchesisselected.Whenaprocess isfinished, controllersets flows onceagain. Inorderto showthe effectivenessoftheproposed scheme,authorsimplementtheEXR andFSR,usinganOpenFlowprotocol.Throughoutthesimulations, authors compare effective utilization ratios of the EXR and FSR. TheresultsshowthatEXRuseslinksalmostwiththefullcapacity; however,FSR onlyreachesabout50% utilizationofits full capac-ity.Asaconclusion,irrespectiveofwhetherthenetworkisidleor busy,EXRenablesenergysavingbycreatingefficientlyutilizedlink structure or full useof capacity.Assigning prioritiesto the flows increases the chance to obtain more flexible management of the network.DelayscanbepreventedusingtheEarliestDeadline Strat-egy,orthroughputcanbe improvedusingshortestjob first strat-egy.Authors observetheaverageutilizationratioofall theactive links through a partial Fat-Tree test bed that containstwo pods and 8 servers. Results show that while the effective link utiliza-tionratiounderEXRalmostreaches100%,itonlyhoversataround 50%mostofthetimewhenusingFSR.Thisresultvalidates energy-centric efficiency of the EXR. However, it should be noted that network resiliency impliesbackuppaths and fastswitchingtimes thatmandatesidleresources.Therefore,theproposedschememay not be able to maintainan acceptable level ofserviceinthe face offaultsandchallengestonormaloperation.
Authors in [35] focusonre-routingthetraffic accordingtothe networkstatus.TheytakeadvantagesofSDNtocreatetheirmodel by collecting real-time informationabout network traffic and in-formation related to users’ requests before joining the network. Thework mainlyfocusesonintegratedchassis2 andline-cards,as
theyconsumeenergymorethananyotherelementinarouter.The goalofthestudyistominimizethelinkusageinsuchawaythat thetotallinkutilizationwillbeunder50%.Inthiscontext,authors symbolizeroutersasstargraphs,wherethecenterrepresentsthe integratedchassisandtheleavesrepresenttheline-cards.Ifthere isnorequest,line-cardsareset tothesleepmodeandifall line-cards are asleep, then the integrated chassis is set to the sleep mode aswell. To validate theefficiency ofthe proposed scheme, authorsrunsimulationsonthreesyntheticnetworksthathave dif-ferentsizesandcomparetheirschemewiththeCplexmethod.The resultsshowthattheproposedschemesavesanimportantamount of energy,outperforming Cplex. However, as depictedby the au-thors,proposedmethodresultsinhighly-loadednetwork environ-ment, which makes the network vulnerable to link failures and suddentrafficbursts.
Authors in [36] presentadifferentapproachforre-routingthe traffic. They demonstrate that existing routing protocols, such as OpenShortestPathFirst(OSPF)orIS-IS [37] spreadtheloadamong multiple paths andachieve QoS. Nevertheless,these protocolsdo not save enough energy, since they increase the total activityof links(turningthemonandoff).Therefore,authorsproposeanew methodcalledRoutingforMinimizationofActiveDevices(RMAD). WhatRMADessentiallydoesistofindanumberofshortestpaths andamongthem,tochoosethepaththathasthemaximum num-ber ofactive links. The aimisto avoidactivatingmore linksand to increase the sleep time of the links. Afterwards, sleep state links are activated along the chosen path. Authors also propose RMAD+, which provides a hop count constraint on paths once
2A router consists of an integrated chassis (includes power supply, heat dissipa- tion system, cables etc.), several line-cards and multiple ports [24].
shortestpathsarefoundandchoosesthepathwiththeleast num-berofactivenodes.Sincenodesconsumemoreenergythanlinks, RMAD+isestimatedtosavemoreenergythanRMADdueto hav-ing lessactive nodes.In order to evaluate the efficiencies of the proposedtwomethods,authorsutilizemulti-rootedfat-tree topol-ogyfortheirsimulationandobservethenodesleepratioand aver-agetransmissiontime.Theresultsillustratethatnodesleepratios ofRMADandRMAD+arehigherthantheratiosofOSPF(RMAD+
isthehighest).RMAD+alsohasthehighestaveragetransmission time since itchooses the path withthe lowest numberofactive nodes,whichisnotnecessarilytheshortestpath.
Flow-controlledSDNsolutionsmainlyprovideaninternal com-munication among network devices to exchange power-related dataandoptimizethechoicebetweenenergysavingandnetwork performanceaccordingtothetrafficload.Sincestationsareaware ofthetrafficloadofthenetwork/rest-of-the-nodes;controllersset an optimized traffic flow by turning off asmany devicesas pos-sible, collecting real-time information about network traffic and informationrelatedtousers’requests beforejoiningthe network. Inthis context,existing flow-controlled SDN approachescan bal-anceQoSandenergyefficiencywithaqualitativenetwork manage-mentsystem [28,30] ,providemultipleoptionsforchangingtraffic load [29] ,enablefair-sharerouting [34] ,supportan optimized en-ergy management during low activity periods [35] , and decrease the numberof active devicesto obtain power saving while hav-ing qualitative communication [36] . Nevertheless, thesesolutions mayalsohave some drawbacks.Forinstance,executionof Open-Flow withGAL mayresult indelay forstations inresponse time [28] ,controller’sresponsetime totrafficchangesmaynotbethat effectiveassuggestedinfastchangingchannelcondition [29] , im-plementation of all three level operations may cause additional overhead [30] ,transitionbetweenactiveandsuspendedflowsmay causeadditional delay as all links work with very high capacity [34] ,energysavingcanbeachievedonlywhenthenetworkis rel-ativelyidle [35] ,oranincreaseinenergyefficiencymayevencause adramaticdecreaseinnetworkthroughputperformance [36] . 4.2. Energy-aware SDN approaches for data center networks
Data Center Networks (DCNs) are another application area of greennetworking.TheriseofIT technologies,suchascloud com-puting, led to an enormous increase in the numberof DCNs. To offerfulltimequalitativeserviceexperience,datacentersare7/24 availableandthissituationresultsinanexcessiveuseofenergy.In 2009,DCNswereaccountedfor2%ofworldwideenergy consump-tionwithan economic impactofUS $30billion [38] . Thisrateis expectedtogrowapaceeveryyearsincedatacenterhardware ex-penditures are growing dramatically [39] . Some of energy-aware DCN-relatedsurveyscanbe foundin [40,41] .AtypicalDCNisthe hometo thousandsofhosts; itstoreshundredsofnetworking el-ements,manyprocessors,powersuppliers,coolingsystems,etc.In orderforthenetworkingpartto workproperly,all oftheseparts shouldbeturnedon [42] .
TodecreasethepowerconsumedbyDCNs,firstweneedto op-timize power consumption of servers and other networking ele-mentssincetheyarethefundamentalsourcesofpower consump-tion [43] .Forinstance,storagerepresents20to40percentoftotal power consumption ina typical data center. It consumes an im-portantamount ofenergyevenifthe device isinthe idlemode. Therefore,optimizingstorageproblemscanimproveenergysaving. Resource sharing between servers, such as memory,storage, and fans, can be another wayof tackling the energy efficiency prob-lem.Anotherapproachistooptimizecomponentsthatdonot con-sumepowerproportionallywiththeirutilization. Theaverage uti-lization of a serveris mostly about10–50% of its maximum uti-lization [44] andmore than 80% ofits linkshave very low
traf-fic [45] . Thus, re-arranging traffic using optimal paths and elim-inatingidle resources by puttingthem into low-energy states or turningthemoff enablesavinghighamountofenergy [46] .Inthe literature,therearenumerousstudiesthataimatreducingthe to-talenergyconsumptioncausedbyDCNs.However,throughoutthis paper,wewillonlyinvestigateenergy-efficientDCNsolutionsthat arebasedonSDN.
ElasticTree [47] isone ofthe mostpopular worksproposedto saveenergyinDCNsusingSDN. Itdynamicallyoptimizesnetwork energy by ensuring qualitative traffic flow, meeting performance constraints,andturningoff asmanydevicesaspossible.Itis com-prisedof three parts: optimizer, routing , and power control . Opti-mizerpartoutputs aminimum-powersubsetofthenetworkfora givenloadusingtheinputs offaulttolerance, powermodel, traf-ficmatrix,andtopology.Then,theresultsare sentto therouting andpowercontrolparts.Routingpartchoosesthepaths forflows andappliestheroutingset.Finally,thepowercontrolpartarranges powerstatesandturnsthedeviceonoroff ifnecessary.
In ElasticTree, the optimizer has three options to enhance the traffic: Formal Model, Greedy Bin-Packing , and Topology- aware Heuristic . Formal Model formulates the problem as Multi- Commodity Flow problem and tries to minimize network power whilesatisfyingall trafficconstraints(e.g.link capacity,flow con-servationanddemandsatisfaction)ofMCFproblem.Formalmodel forcestraffic to flowthrough active linksand switches.However, itsscalabilityislow,andittakestimetocalculatetheoptimalset. Therefore,itisusefulfornetworkswithlessthan1000nodes.The second option, which is Greedy Bin-Packing, does not distribute theflowrandomly;instead,itchoosestheleft-mostpath.Ifaflow cannot be transferred usinganypath, it willbe sent to the path with the lowest load. Finally, Topology-aware Heuristic uses ad-vantagesoffat-treetopologytomanagethetraffic.Itassumes per-fectlydivisibleflows;itsplitsthedataaccordingtothelink capac-itiesand activates therequired linksin linewiththe demand of aboveorbelowlayer. Perfectlydivisibledataisnotalways practi-calandmayleadlowqualitysolutions;yettheproposedheuristic methodisfasterthantheothersanditrequireslessinformationto work.
ECODANE [48] isanotherprojectthataimstoobtainenergy ef-ficiencyin DCNs. It has five modules: Data center network, Opti- mizer, Power control, Forwarding ,and Traffic generator .Authors use the ElasticTree [47] data center network, which is designed for Energy-optimizedData Centers. The Optimizer module computes minimumenergy subset usingthe Topology-aware heuristic [47] . The Power control module forces switches to turn on/off or to changethe state to an appropriate powermode. The Forwarding moduleuses ahierarchicalload balancingrouting module,which workstogetherwiththeoptimizer.Throughoutthepaper,authors performextensiveteststoobservetheefficiencyoftheirproposed method and obtain 10% - 35% energy saving depending on the trafficlocality,whetherit ishighlylocalized(withinarack), mid-localized(withinaPOD),ornon-localized(acrossthedatacenter). A hybrid data centertestbed, usingthe idea behindthe ECO-DANEproject, isproposed in [49] .The aimofthe workis to ob-tainahybridtestbedusingrealnetworkelementsandvirtual em-ulationtools,to have an OpenFlowcontroller supporting energy-aware functions, and to better understand which parts of the networking elementsconsumehow muchenergy. Hybridtestbed helpstoobtain scalable andrealistic network experience sinceit combinesphysical andvirtual tools.In thiswork, network archi-tectureconsistsofadatacenternetwork,an OpenFlowcontroller, anda traffic generator. NetFPGA [50] routers, which allow mea-surementsof power consumption andregulations of power scal-ing, are used as the hardware part of the network. Additionally, Mininet [51] ,asimpleandlow-pricednetworktestbedtodevelop OpenFlowapplications,isusedfortheemulation.
Authorsin [52] proposeECDC,whichisanenergy-efficientdata centerapproach.ItisanOpenFlow-baseddatacenterscenario hav-ing different types of customers, such as private home users or business users.ECDC separatesthe traffic accordingto users’ de-mands.ThenetworkiscomposedofOpenFlowcontroller,switches, servers, anda management stationthat isconnected to allother devicesinthenetwork.Managementstationorchestratesthe net-work, generatespolicies,allocates virtual machines,andturns off unused devicesby checkingpredefined thresholdvaluesor time-outsofeach services. Aslong asmonitoringinformation onCPU, power,andmemory-loadisavailableandprovidedtothe manage-mentstation,ECDCenablessavinganimportantamountofenergy. In [53] ,authors propose a Correlation-Aware Power Optimiza-tionAlgorithm(CARPO),whichconsolidatestrafficflows by elimi-nating unnecessarylinkstodecrease energyconsumption.CARPO focusesonsavingtheenergyofDCNratherthanloweringservers’ energy consumption. In this work, authors implement the algo-rithm using a central power manager and OpenFlow switches. Sincetheswitchesconsumehighpercentageoftotalenergyofthe networkevenwhentheyareinidlemode,turningthemoff isthe bestwayto lowerthe energyconsumption.Using theconstraints ofmaximumlink capacityandthe equalityofincomingand out-going datarate,consolidation proceduretriesto minimizeenergy consumptionbyturningoff switchesasmuchaspossible.
The difference of CARPO fromthe existing approaches is that CARPO rearranges the traffic flow according to the correlations between flows. In [53] , authors validate two main observations. First, different traffic flows are not tightly correlated orthey are sometimeseven negativelycorrelated,so theydonot peakatthe sametime. Second,90% ofthe linkutilizationsis muchlessthan thepeakvaluesoftheflow, sotheconsolidationprocess mustbe basedontheaveragelinkutilizationotherthanpeakvaluestosave moreenergy.Inthiscontext,CARPOfirstanalyzescorrelations be-tween flows,andthenconsolidatesthe trafficaccordingly.Finally, thealgorithmsetslinkratesaccordingtothedemand.Theauthors solve the problemusingmathematical tools atfirst andobtain a near-optimalsolution,whichrequiresperfectknowledgeoffuture demand.Toobtainareal-lifesolutionfortheproblem,theauthors alsointroduceanalgorithm basedongreedy-binpacking.This al-gorithm assigns links to the flows, while satisfying link capacity andcorrelation requirementsbetween concurrent flows. In order to show the effectiveness of the proposed scheme, authors also comparethe energyconsumption andtheperformance ofCARPO withthe ElasticTree [47] ,GoogleP [44] and CARPO-C,which is a version of CARPO without link rate adaptation. In this context, authors implement a hardware test-bed that is composed of 10 virtual switches configured with a production 48-port OpenFlow switchand8servers.TheempiricalresultsshowthatCARPOleads tohighamountof energysaving (upto46%)anda limiteddelay increaseforaDCN.
Consequently,utilizing SDNindata centernetworksmay lead to several energy-centric advantageous, such as obtaining quali-tative performance,energy efficiencyandrobustnessatthe same time [47] ,adaptingthesetofactivenetworkcomponents dynam-icallyaccordingtothetotaltraffic [48] ,balancingtheQoSand ef-ficientuseofresources [52] ,andfocusingonsavingnetwork’s en-ergy other than server’s deeplyanalyzed traffic [53] .Yet, the ex-isting proposals mayalso have some drawbacks. Forinstance, in [47] ,findinganoptimalsetofactivelinksandtrafficflowmaytake longerthanpredictedandanyadaptationattemptmayfailormay causeaperformancelossinfastchangingchannels.In [48] ,overall throughputperformance mayevendecreasesince thereisno ad-justmentforswitches’ link rates.In [49] , DCtestbed architecture maycausesensitivedatatobetransferredlate.In [50] ,using out-datedinformationmayresultindramaticdecreaseonperformance and energy saving since monitoring information on CPU, power,
and memory-load may not be always available. Besides, in [53] , highly loaded links may cause delay-sensitive multimedia flows sufferastheprocedureincreasesdelay.
4.3. Rule-placement and TCAM-based energy-aware SDN approaches Authors in [54] proposea methodto eliminate therule space problem of existing energy-aware routing (EAR) approaches.EAR approachesmainlyassumethatthereisaninfiniteamountofrule space andhence an OpenFlowswitch can hold an infinite num-ber ofrules.However, thisassumption isnot realisticin practice, asflowtablesare implementedwithTernaryContentAddressable Memory (TCAM), which isexpensive andpower-hungry. Tosolve the problem, authors come up with the idea of default rule . Ac-cordingtothiswork, ina routingprocess,ifthereisnot any pre-definedruleforapacket,thedefaultruleisapplied.Toavoid com-municationdelaysbetweenroutersandthecentralized controller, packets thathavethedefaultruleareforwardedonlytoa default port(eachswitchhasonlyonedefaultport)withoutcontactingthe controller.Inthiscontext,authorsfirstformulatetheproblem us-ingIntegerLinearProgram(ILP)andpresentaheuristicalgorithm to minimize energy consumption. The algorithm consists of two steps.First,itfindsfeasibleroutingsolutionforeachrequest, pro-vidingcapacityandrulespaceconstraints.Theflow tableisfilled until it gets full. Subsequently, the port that carries mostof the flow is assigned as the default port so that there will be more space available for new rules. Second, it reduces the number of activelinks;thelinkswithlowtrafficloadareturnedoff andtheir trafficistransportedtootheractivelinks.Animportantportionof energycanbesavedusingtheproposed methodasitreducesthe amount ofrulespace, communicationdelay betweenroutersand thecontroller,andthetotalnumberofactivelinks.
Anothermethodthataimstodecreasethepowerconsumedby TCAMistheCompactTCAMmethodproposedin [55] .Sincelookup process consumeshighamountofenergywhilesearchingfor356 bits3 flow entries inTCAM flow table, the authors introduce the
Flow-ID concept in lieu of standard flow entries in order to de-creasethepowerconsumptionandcost.InthenewTCAMlookup process,foraflowtobe forwarded,acompletematchintheflow table isnot necessary;asmallermatchisalsoenough.Thus,flow entries can be compacted into smaller IDsand switches can ap-plyallprocesseswiththisnewID.TheFlow-IDassignmentprocess startsonceanewflowentersthenetwork.Ingressswitchforwards apackettothecontrolleriftheflowisnotfoundintheflowtable. ThecontrollerformsaFlow-ID,storesitinitslocalreferencetable, andthenre-sendstotheswitchcontainingsomeinformationwith aCOMPACTflag.Iftheflagisclear,therulesonthepacketare pro-cessed.Otherwise,theactionsbasedonFlow-IDareperformed.At thefinalstep,thecontrollergivesacommandtoremovethe Flow-IDandthentheoriginalpacketisreturnedandforwarded.Authors buildtheproposedCompactTCAMframeworkonanOpenFlow en-abledNetFGPAplatformanddemonstratethatswitchfabricpower gain canbeabout2.5timesofa layer_2switchandhighamount of energy saving (up to 80%) can be achieved compared to SDN switches.
IthashithertobeenclearthatthecentralizednatureoftheSDN controlleranditsabilitytofolloweverysingleflowinthenetwork isbeneficialfornetworkmanagement.Authorsin [56] propose De-voFlow,whichisamodificationoftheOpenFlowcontroller.It sim-ply breaks the couplingbetween centralized control and central-izedvisibility;inotherwords,itisaprotocolthatdisburdensthe controller by assigning some of its work to switches and obtain
3In SDN, flow entries can potentially support large number of flows, such as 15 field tuple - that requires 356 bits to define a flow.
cost-effective and energy efficient management of the network. Theunderlying reasonforthismodification isthe interferenceof thecontroller in flows transfers highamount ofworkload to the controlplane.Besides,collectingstatistical(periodical)information allaroundthenetworkburdensthecontrollerandlessensits abil-itytomanagebigpackets.Toeliminatethesedeficiencies, authors developsomemechanismstoincreasethecontrolofswitchesand the efficiency of statistics collection. Rule Cloning is the action whereifaflow matcheswithanotherflow,theswitch clonesthe rulelocallytopreventunnecessary useofTCAMandtominimize communication betweencontroller andswitch. In addition, with theproposedLocalActionsmechanism,switchescanmanagesome setofflowsontheirownwithoutinvokingthecontroller. Further-more,to improve theefficiency ofOpenFlow statistics collection, authorsoffertwodifferentapproaches;(i)UsesFlowand(ii) Trig-gers andReports. Use sFlow, which is implemented in a switch, collects information that contains headers of randomly chosen packets via sampling and transmits it to the controller. Triggers andReportssendsstatisticalinformationwithapush-based mech-anism only when a triggering condition is met. Accordingly, au-thors’custom-madesimulationresultsshowthatDevoFlowcan in-creasethroughputoverECMProutingby16%and24%forClosand HyperXnetworks,respectively.
Authorsin [57] alsodiscussTCAMusedinOpenFlowswitches. Tolowerthepowerconsumptionandlatency,theyproposeanew OpenFlowswitch.Inthiswork,whenapacketisreceived,itis di-rectedto the input memory. While receiving it, a module called PacketParserextracts its importantfieldsandcreates aflow-key, which identifies the packet. Flow-keys are stored and searched inflow tables that can be implemented insoftware-based SRAM orhardware-based TCAM. Inaddition, to lower the time used in TCAM,apacket predictioncircuitryis alsoimplemented. The cir-cuitry triesto predict the flow-key andprevents the flow-key to entertheTCAMlookupprocess.Thepaperpresentstwoprediction methods. One is Direct Map, which extracts flow-keys from pre-definedlocation. Thismethod selectsthe mostvarying fields be-tweensubsequentflows.Thesecond oneiscalledSub-FieldHash. Itintelligentlyextractsflow-keysfrompreciselydefinedfieldsand uses DJB Hash function to form a packet signature. Prediction circuitryachieves both energy saving andlow latency, bypassing TCAM.Energy savingofswitcheswithpredictioncircuitrywillbe highenoughandenergywastewillbeinsignificant ifcorrect pre-dictionratesarerelativelyhigh.
Consequently, Rule-placement and TCAM-based SDN ap-proaches may enable an energy-aware routing method while optimizing rule spaces of switches [54] , reduce the size of flow entries and manage large sized SDN flows [55] , ease the con-troller’s work bytransferring some of its worksto switches [56] , andlessentheeffectofTCAMonpowerconsumption [57] . Never-theless,fast-changingdynamicchannelsmayalsocauseswitching thedefaultporttoofrequently [54] ,formingaFlow-ID,storingand re-sending it to the switch may also increase latency [55] , local actions takenby switches complicate thecontroller’s globalview andmanagement [56] ,andfast-changingchannelsmayreducethe correctpredictionrates [57] .
4.4. VM placement-based energy-aware SDN approaches
Energy Efficient and QoS aware Virtual Machine (VM) Place-ment (EQVMP) [58] is a method that aims to cure lack of en-ergy saving mechanisms of VM placement. Energy-efficient algo-rithms in VM placement policies sacrifice from network perfor-mance. After aggressively cutting down the energy usage, tradi-tional routing algorithms fail to find optimized routes forflows. SDNcanovercomethisissuebyimplementingeffectiverouting al-gorithmsandappropriate VM allocations inorder toachieve low
latencyandhigh throughput,i.e.a qualitative network. To main-tain a desirednetwork performance, authors propose EQVMP al-gorithm,whichconsistsofthreesteps:VMsresourcedemand,VM traffic,andtopologymatrix.InEQVMP,HopReductionpartgets ac-tiveatfirst.VMsandthetrafficloadbetweenVMsarerepresented asa graph.Toreduce thenumberofhops, thegraph-partitioning problemismodeled. Inthiscontext,thegraphispartitioned into smallergroups withfewer connections. This way,Hop Reduction reducesthecostofrelationbetweengroupsandincreasesthe bal-anceofeachgroup.Afterreducingthetrafficloadbetweengroups andloweringthedelaywithshorterconnections (HopReduction), EnergySavingpartminimizestheresourceutilization.Aftersorting VMs accordingto the resource demands in decreasing order,the algorithmseeksaVMwithminimumresourcesavailablefora re-centlyactivatedserver.Finally,thelast stepistheLoadBalancing. Thismoduleprevents congestionwiththehelp ofSDNstructure, andre-routesflowsusingthecontroller.Iflinkutilizationreaches apre-determinedthresholdvalue,flowsareheadedtoanotherlink thathas lessutilization. Inorder tovalidate the efficiencyof the proposedscheme,authorsbuildanNs-2basedsimulation environ-ment that has a 3-tier fat-tree datacenter network, consisting of 16core-level,32aggregate-leveland32edge-levelswitches.Each edge-levelswitch canconnect8servers,andeachservercanhost 4VMs. EQVMP iscompared to well-known VM allocation meth-ods,such asFirstComeFirstServe(FCFS),Largest TaskFirst(LTF), andRoundRobin(RR).Theresultsshowthat EQVMPoutperforms otherallocation methods andenhancessystemthroughputby al-most25%.
Authors in [59] focusontraffic-awareplacementofVMsto ob-tain better utilizationof resources. Theypresent two algorithms: server-driven and network-driven . These algorithms aim to effi-cientlyallocateresourcesofintra-DCN.Theserver-drivenalgorithm firstchoosesaserver fora VMandthendeterminestheswitches thatwillprovidetheflowoftraffic.Thenetwork-drivenalgorithm firstselectstheswitchesandthenanappropriateserverforVM.To selectservers andswitches,therearethree options:FirstFit (FF), BestFit(BF),andWorstFit(WF).FFranksserversandswitchesin ascendingorderaccordingtotheircomputationalpowerand over-alltraffic load, respectively,andthenchooses thefirst onein the list. WFchooses the most unloaded servers or switches. Finally, BFchoosesthemostloadedserversorswitches.Inordertoshow theeffectivenessoftheproposedmethod,authorsevaluatethe re-sultsobtainedinsimulationintermsoflinkutilization.Theresults showthatnetwork-drivenalgorithmwithFFoptionleadstohigher powersavingthantheothers.
Authors in [60] combineVM placementandrouting optimiza-tionin DCNs to achieve energysaving andpropose a joint host-networkalgorithmbyusingdepth-firstsearchwithbest-fitoption andselectinganelementwiththesmallestandsufficient remain-ing capacity. Authors allocate particular nodes for each VM and another dummy node for each connection between the VM and servertoindicate thememoryconstraintof theserver. Theyalso introduceaparallelprocessingmethodthatdividesDCNsinto clus-tersandfinds optimalpaths ina parallel way.The methoddeals withconnectionsinbetweensubnetsofserversandVMs.Authors’ custom-made simulation results demonstrate that the proposed schemeoutperformsexisting network-only and host-only optimiza-tionsolutionsintermsofenergyefficiencyandthroughput.
In a nutshell, VM-placement based SDN approaches mainly combinetwo effectivemethodsto saveenergy;(i) VMplacement and(ii)routingoptimization [60] .Theseapproachesmayresultin notonlyenergysaving,butalsoachievehighthroughputandlow latency [58] , providecloud-fluenttraffic engineeringandincrease in link utilization [59] . Nevertheless, these approaches may also havesomedrawbacks;re-computationtimeofnewVM-placement mightbehigherthan predicted [58] ,proposed scheme(s)maybe
toolate to react in fast-changingchannels [59] ,andoperation of VMmigrationitselfmayconsumehighenergy [60] .
4.5. Different types of energy-aware SDN approaches
Intheliterature, mostoftheproposed energy-aware network-ing solutions make use of Layer 3. However, SDN can also pro-videefficiencyinlowerlayers.Asanexample,authorsin [61] pro-pose a controller that makes use of Layer 2 protocols to reduce theenergyconsumption. Maingoalofthe proposed schemeisto prevent connections that causeloopsbetween OpenFlow-enabled elements.The controller in this work is called GreenMST. It uses minimumspanningtreeprotocol(STP)topreventexcessiveenergy usage.The control softwarewasbuiltupon theBeacon controller [62] ,whichisanopen-source,multithreading-supported,and Java-based OpenFlowcontroller. GreenMST storesinstantaneous states ofthenetworkusingTopologyManager(TM).TMusesLink Layer DiscoveryProtocoltofigureoutwhetherthereare anychangesin the connectivityof switches. Incase ofa change, newminimum spanningtreeofthenetworkiscalculatedusingKruskalalgorithm. Afterwards,theproposed schemecreatesaChange Set,which in-formsthecontrolleraboutthelinksthatwillbeturned onoroff. Inorderto examinewhetherthesystemis workingproperly, au-thorsmainly study threedifferent periods duringthe tests. First, theyexaminetheperiodstartingfromtheactivationofGreenMST untilreachingtoastableconditionsothatGreenMSTrulesare en-sured(no loops, nopartition, etc.).Second, they examinethe pe-riod when a new device enters the network until the controller updatestheoverlayofthenetwork.Finally,thethirdstudycovers theperiodofcalculatingMSTandcheckingtheconsistencyofthe networkwithGreenMSTsettingsagain.GreenMSTcansavean im-portantamountofenergyasitprovides loop-freecommunication amongdevices.However,theproposedsystemdoesnotdetect net-workfailures,suchasalink-down.Besides,itmaynotworkwellin largenetworkssincethetime measurements revealthatthe time spent doesnotgrowlinearlywiththeincreasing amountoflinks betweendevices.
Authors in [63] introduce a Hyper-Cellular network protocol calledHyCell, whichis an exampleofgreen software-defined ra-dio access networks (SDRAN) with energy efficient base station (BS)operations.HyCelldecouplescontrolandtrafficcoverage,and managesthemusingBSs,ControlBSs(CBSs),andTrafficBSs(TBSs). CBSsprovidecentralcontroloverTBSsandmobileusers.CBSsare responsibleforuserequipment’s(UE)networkaccess.Inthis con-text,CBSsassignUEstothenearestandavailableTBSs.HyCellalso decouples softwareandhardwareofBSs inorderto provide eas-ierupdatesforBSs.ActiveTBSssendtheir trafficloadinformation toCBSs,whichthen determinetheleastloadedTBSforawaiting user.Since CBSshave theglobalview ofTBSs, they canput TBSs intosleepmodeiftheyhavealowtrafficloadtomaximizeenergy efficiency.AuthorsalsopointoutthatthewakeuptimeofPCsare high.Therefore,theproposedmethodmaynotbesuitablefor sce-nariosrequiringlowlatency.
Authorsin [64] implementanalgorithmtoreduceenergy con-sumptioninPassiveOpticalNetworks(PON),whicharefiberoptic access networks. According to the statistics, access networks are responsibleforalmost70%ofthetotalenergyconsumptionofthe Internet [65] .PONsystemsalsoconsumehighamountofenergyin theInternettoday.InaPON,therearemainlytwomodules: Opti-cal LineTerminal(OLT) andOptical NetworkUnit(ONU).Authors createan EMM-UDS algorithm to reduce energyconsumption by defining three energy states forboth OLT andONU: awake, port sleep , and board sleep for the OLT, and awake, sleep , and critical forthe ONU. Theproposed algorithmis initiatedwhen the max-imum time required to complete a data transmission is smaller thanthepre-determinedthresholdvalue.Oncethetransmissionis
completed, ONU sends queuelength information tothe OLT, and then OLT sends thisinformation to theOpenFlow controller. The controllerdecidesonthesleepconditionsofONUs.Ifthecontroller decidestoputONUintosleepmode,thecorrespondingOLTportis alsosettoportsleepmode.Afteracertainperiod,ONUentersthe criticalmodeandOLTportactivatestheawakemodetowarnthe controlleraboutthebufferqueuelength.Ifthecontrollerwakesup theONU,thepowerstateofONUisalsochangedtoawakemode.
PayLess proposed in [66] is a monitoring framework that de-creases the cost of collecting statistical information. In SDN, ap-plicationscollectstatisticsviacontrollerwithdifferentperiods,i.e. per flow, oncein five second, etc. Collectingthis information in-creasestheburdenonthecontrollerandcausesnetworkoverhead. PayLessprovidesaseparatelayertocontrolthecollectionof statis-tics. It consistsoffour components; Request Interpreter, Scheduler, Switch Selector , and Aggregator & Data Store . Request Interpreter provides communicationbetweenlayers. Schedulermanagestime forcollectingstatistics.SwitchSelectorchooses theswitchesfrom which theinformationwill be extracted. Finally,datais collected and stored by the Aggregator & Data Store component. Applica-tions request statistics using RESTful API of PayLess. Users then create a Monitoring-Request object, specifying Type,Metrics, En-tity, AggregationLevel,Priority, Monitor,andLogginginformation aboutthe requestedmonitoring object.PayLessresponds withan access-id,andthenapplicationusesthisidtoextractrequested in-formation.Althoughenergyefficiencyisnotthemaingoalof Pay-Less, statistical information obtained through it can also help to achieveanenergy-efficientnetworkingenvironment.
Authors in [67] propose an extension for OpenFlow switches to make them work in different power saving modes. In this work, Power Control module, referred to as OpenFlow Switch Controller (OSC), sends the power changing commands. Con-troller employsOSCtomanagepowerstatesofswitchesfor turn-ing on or off a switch, disabling or enabling links or ports, or changing energy consumption level. They introduce three types of new messages (OFPT_PORT_MOD, OFPT_LINECARD_MOD and OFPT_SWITCH_MOD) to change the energy states of ports, line cards,andswitches.Accordingly,authorsbuildahardwaretest-bed includingaNOXController,anOSCandaNetFPGAbasedOpenFlow switch.Experimentalresultsshowthattheproposedextensioncan sendcommandsfromtheNOXtotheOSCtoturnon/off theswitch andtoturnon/off theportsoftheNetFPGAcard.Inaddition,total powerconsumption(1900mW)costoftheextensionisnegligible, whenthepotentialofhighamountofpowersavingisconsidered. Authors in [68] emphasize that currentlySDNs usually coexist with traditional networks, andhence they are partially deployed atpresent.Inthiscontext,authorsinvestigatehowtosaveenergy inpartiallydeployedSDNs.First,they formulatetheproblemasa multi-commodityflowproblemusingconstraintsoncapacity,flow, etc.tominimizethetotalnumberofSDNdevices.Sincethe formu-lationis notpractical duetoits complexity,a heuristicalgorithm followstheir formulation.Thealgorithmfirst obtainsthesmallest closed sets; nodesin thissetonly havetraffic with thenodes of thisset.Thealgorithmgivesweightstothelinksandnodesin or-dertocalculatetheenergyconsumptionofeachpossiblepath. Ac-cording tothe traffic demand, thealgorithm then finds spanning treesforthesmallestclosedsets tofigureout thepathwith min-imum energy.The purpose hereisto minimize the distance.Ifa traffic flow betweena set of nodesis chosen, thealgorithm first chooses an edge between these nodes with the smallest weight andcontinuesthisprocessuntilitreachesthedestination.Ifthere is an overloaded link,then the algorithm removesit andrepeats thesamestepsusingthesecondhighestoption.
Accordingly, the proposed SDN controller approach [61] that mostlymakes useofLayer 2protocols mayassist avoiding loops in-between connections. However, in theproposed scheme,
com-putationtime increasesfasterthan the numberof linksbetween devices,which can be a problemin large networks. Energy effi-ciencycan alsobe increasedinHyper-Cellularnetworkswiththe helpofSDN,yettheproposedmethod [63] maynotbesuitablefor scenarios requiringlow latency.Additionally,PayLess proposed in [66] collects statisticsandeases theburdenonthecontroller, yet itis not compatiblewithdistributedcontroller platforms. Finally, theproposalfortheextensionofOpenFlowswitches [67] provides detailedcontroloverlinecards’andports’energyconsumption.In thiswork, energyefficiency can be further improvedwith a dy-namiclinkrateadaptation.
5. Evaluation of energy saving techniques and proposed approaches ın SDN
Energy saving in SDN can be addressed through hardware or software-based enhancements; in other words, energy efficiency canbeachievedinchip,node ornetworklevels.While hardware-based solutions are mainly applied on forwarding switches, software-basedsolutionsareappliedonthecontrolleroronaccess nodes.Linkrateadaptation,loadbalancingamonglinks,re-routing the traffic flow, turning a device (or only a part ofit) on oroff, ruleplacement,minimizing theTCAM,networkvirtualization,etc. arefundamentaltechniquesthataddressenergysavinginSDN.
Traffic-aware solutions, such aslink rate adaptation,load bal-ancingamonglinks,andre-routingthetrafficflow,areinspiredby thefact thatnetwork componentsare oftenunderutilized. There-fore,turning onoroff networkdevices(or onlyapartofit,such asaforwardingswitch, aCPU,anda port)dynamicallyaccording tonetworkconditionsandtrafficloadenablessavinganimportant amountofenergy. Ruleplacement isanotherwayto saveenergy inSDN. It mainly focuseson how to placethe rules in forward-ing switches, and also on how many rules (a forwarding switch canholdafinitenumberofrules)mustbedefinedtoensurethat similarflowsfollowsimilarpaths.Inaddition,forwardingswitches inSDNuseTCAM.SinceTCAMisvery expensiveandpower hun-gry,TCAMapproachesproposedintheliteratureaimtoreducethe memory need of information stored in the forwarding switches. Lastbutnot theleast,ina virtualizednetwork,physicalelements cancontainmorethanonevirtualnetworkstructure.Therefore,it mayreduce both the cost of constructing newhardware for the newnetworkandpowerconsumedbythishardware.Network vir-tualizationalso provides dynamicnetwork abilities;in casethere is an immediate need fora change in a network, it enables fast transmissionfromtheoldsystemtoanewone [9] .Thiscapability ofnetwork virtualizationalsoresultsinperformance increaseand energysaving.
Alloftheenergy-efficientSDNapproachesexaminedinSection IV evaluate the effectiveness of their approaches by implement-ingtheirownsimulationenvironment.Theparameters,topologies, networkcondition,andthesimulationsoftwareusedbythese ap-proachesare completelydifferentfromeach otherandmainly no comparisonismadebetweenexistingapproaches.Authors simply comparetheefficiencyoftheiralgorithmswiththetraditional net-workingscenario.Therefore,itisextremelyhard tocomparetheir efficienciesintermsofenergysavingwithoutimplementingthese solutionsonebyoneinthesameplatform.
Throughoutthissection, weevaluatethe expectedenergy effi-ciencylevelsofeachapproachindividually.Theevaluationcriteria arecarriedoutbytakingintoconsiderationthespecificmetricsof each approach, such asthe target environment (fixed, mobile or data center),maximum achievable throughput, coverage, number ofactivelinks/devices,impactofeachparametersused,the topol-ogy used in simulations, achieved results, static/dynamic struc-ture of the implemented network, advantageous/disadvantageous oftheproposedscheme,issuesthatmayarise,overheadofeach
is-sue,impactoftheoperation(chip-level,node-level,network-level), additionalprotocolsupports,advantageous/disadvantageousofthe additionalprotocolsupport,etc.Weightbatchingofthe aforemen-tionedmetrics, which are investigated in Section 4 in detail, we believethat itwill alsobe possibleto reachto ageneralopinion abouttheseproposed works.Inthiscontext, inorderto summa-rizethe features,differences andtheexpected amountof energy gainsofthe proposed energy-efficient SDN solutions,we make a briefcomparisonandclarifytheissuesthatmayarisein Table 1 .
As shown in Table 1 , expected energy gain is classified into threelevels;(i) High ,(ii) Medium ,and(iii) Low. High impliesthat theproposed scheme isexpected tosave highamountof energy togetherwithathroughputincreaseoratleastwithouta through-putloss. Medium impliesthattheproposedschemeisexpectedto achievehighorconsiderableamount ofenergyattheexpense of limitedthroughputdecrease.Finally, Low impliesthattheproposed schemeisexpectedtosaveonlyalimitedamountofenergywhile havingdecreaseonthroughputaswell.
Apartfromtheenergyefficiencylevels,proposedSDNsolutions aremostlybasedonasingleenergysavingtechniqueasshownin Table 1 . However, thesetechniques are mostlyindependent from eachotherandcanbeappliedall togethertofurtherincrease the energyefficiency.
6. Issues, challenges and future research directions
Although SDN architecture has been introduced as a trou-bleshooter, italso brings some new issuesthat need to be regu-lated.Forinstance,theriskofattackstothenetworkmayincrease withtheSDNduetodecouplingofthecontrolanddataplane. Re-centlyintroduced elements, suchas controllers-software, control-dataandcontrol-applicationcommunications,can facethreats. In addition,thereisalackofSDNsecurityapplications,whichshould provide a secure transmission between control and application layerthroughNorthboundAPI.
Since therearemultiplevendorsandoperators inthenetwork industry,coexistenceofnetwork components isanother issue.To createanSDNstructure,independentcomponentsneedtobe com-posed;hence,network elementsshould beinter-operable. In this kindofmulti-vendor industry,thereshouldbe astandard forthe inter-communication of network devices. SDN has achieved the creation of a dynamic network structure by facilitating users to controlthenetworkvia applications;nevertheless, thenumberof applications to manage and regulate the network is insufficient. Therefore,additionaleffortmustbeperformedinthisdirection [7] . As mentionedthrough Section 4 ,turninganetwork (ora part ofit) on/off dynamically based on channel utilizationmay result inenergysaving [29,47,52-54,67] .Yet,itischallenging,anNP-hard problem [69] ,asthereisatrade-off amongenergyefficiency,delay andthroughput performance. Energy-efficient SDN paradigm has beenexamined verylittle fromthe theoreticalperspective inthe literature.As futureresearch,optimizationproblemsthat focuson thesetrade-offscanbestudied.Theseproblemsmaythenbe ana-lyzedfromtheperspectiveofapproximationalgorithmsor param-eterized complexity. Dynamic networks constitute an even more challengingenvironmentforoptimizationproblemsandhence, ap-plyingheuristicsolutionswithlowcomplexitywhilealso address-ing these trade-offs has paramount importance in dynamic net-works.
Anotherissueis theoptimizationofthememorysize usedby forwarding switches as flow tables are stored in TCAM [54-57] . However,TCAMalreadyprovidesalimitedmemory.Ifitisreduced evenfurther,itwillcausefrequentflowentryreplacementswhen new flows are to be installed. Frequent flow entry replacement meansmoreaccessestoTCAMandhence,moreenergy consump-tion. Optimization problems addressing these tradeoffs can be a
fruitfulavenueforresearch.Forinstance,oneproblemmightbeto optimizethememorysizeusedbyforwardingswitchesinorderto minimizetheenergyconsumptionsubjecttoacertainupperlimit forflowentryreplacements.
AnotherissueinSDNistheruleplacement [54,56,62] .Ithasa directimpactonrouting,andhenceonnetworkperformanceand energyefficiency.SDNcontrollerhastoplaceenergy-aware adap-tive rules into switches under some constraints, such asnumber ofswitchesandrouting policies.Thisissuehasnotbeenexplored thoroughly in the literature. Novel traffic-aware and repetition-basedrule placementmethods thataddress thetradeoff between routing performance and energyefficiency in the SDN controller needtobedesigned.
Flexibility,whichisthecapabilityofthesystemtoadaptto dy-namicnetworksettings, andscalability,which isthecapabilityof anetwork tohandleagrowing amountofwork,are alsotwo es-sentialresearchissues.Traffic-awareSDNapproachesarerequired to be flexible [34] and scalable [49] . Although there have been manyenergy-efficientSDNrelatedproposalsintheliterature,they donot mainlyfocusonscalabilityandflexibilityissues stemming fromdense network deployment,channel fluctuations and differ-ent nested topologies. Existing works can be explored in terms of horizontal (adding/removing more nodes to/from a network) andvertical (adding/removingresources to/from a singlenode in the network)scalability.Additionally,flexibility ratios ofdifferent nested topologiescan be compared andmore scalable and flexi-blenovelsolutionsthatcanbechangeadaptivelyaccordingto dy-namicchannelconditionscanbeproposed.
Fairnessisanother issuethat should be takenintoaccount in communicationnetworksespeciallyinresourceallocation.Fair al-locationofresourcesisusuallydesired;yet,thisrequirementmay conflict withother objectives, such astotal data rateandenergy consumption. Fairness is vital in many scenarios such as band-widthallocation,traffic engineering,Quality ofService(QoS),and energy consumption. Since fairness is a global measure as it is related to the resource allocation of all entities, global network knowledge and centralized decision-making mechanism of SDN make it convenient to provide fairness [34] . There are also cer-tain limitations of SDN that lead to new fairness measures. For instance,SDNhaslimitedflow table sizes;therefore,resource al-location mechanisms need to ensurefairness to the network en-tities in terms of flow evictions. A possible research direction is todesign resourceallocationmethods providingfairnessinterms offlowevictionsandflowtablesizesinSDN. Inaddition,fairness inenergyefficiencyislargelyunexploredinSDNliterature.When therearemultipleSDNdomains,fairnessintheenergy consump-tionofthesedomainsshouldbetakenintoaccountwhilemaking resourceallocationdecisionsrelatedtorouting,trafficengineering etc.
Networkvirtualizationisalsoaneffectivewaytoovercomethe ossificationofInternetbyenablingmultiplevirtualnetworksto co-exist ona shared infrastructure [58–60] . Virtualnetwork embed-ding/assignmentis a resourceallocation problemconcerned with the assignment of physical resources to the virtual networks. In thiscontext,virtualizationofserverscanbeanotherwaytoreduce energyconsumption inSDN. As an example,authorsin [70] sug-gest using SDNfor virtual network embedding due to its global topologyknowledge.Thisway,multiplevirtualmachinescanwork on thesamephysical server. Thus,instead ofusingmany servers inefficiently, constructingvirtual machinesenables additional en-ergysaving.Apossible researchissueisto implementSDN-based routingalgorithmswithappropriateVMallocationsthatreducethe total number of hops, minimize resource utilization, balance the traffic loadandachieve highthroughput, low latencyandenergy efficiency.
