LTE Test Case App Note

(1)

WEBSITE:

WEBSITE:www.jdsu.com/testwww.jdsu.com/test

LTE and EPC Test

An Overview of Test Concepts and Tools for Trials

Speci

Specic LTE Tc LTE Test Areas est Areas � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �3� � � � � � � �3

Overview������������

Overview��������������������������������������������������������������������������������������������������������������������������������������������3������3

Gainingdeeper

Gainingdeeperinsights���insights��������������������������������������������������������������������������������������������������������������������������������4���4

Performancetesting

Performancetesting�����������������������������������������������������������������������������������������������������������������������������������5���������5

Single-userth

Single-userthroughputroughputtesting�����������testing�������������������������������������������������������������������������������������������������������5����5

Cellperformance

Cellperformanceverificationwithverificationwithmulti-userthmulti-userthroughputroughputtestingtesting�������������������������������������������������������6���6

Realisticmulti-user

Realisticmulti-userthroughputthroughputtesting��testing������������������������������������������������������������������������������������������������8������8

Idle-to-activetransition

Idle-to-activetransitiontimestimes�����������������������������������������������������������������������������������������������������������������9�������9

Latencytesting��

Latencytesting����������������������������������������������������������������������������������������������������������������������������������������������1010

KPIverification

KPIverificationandcalculationandcalculation����������������������������������������������������������������������������������������������������������������10������10

ValidatingLTE

ValidatingLTEvoicevoice����������������������������������������������������������������������������������������������������������������������������������������1212

Signalingvalidation���������������

Signalingvalidation�����������������������������������������������������������������������������������������������������������������������1313

VoiceQoS

VoiceQoSandQoEandQoE(MOS)(MOS)������������������������������������������������������������������������������������������������������������������������1313

TestingQoS

TestingQoSandQoEandQoEofLTEofLTEstreamingvideo����������streamingvideo����������������������������������������������������������������������������������14��14

EvaluatingLTE

EvaluatingLTEMIMOandMIMOandfrequency-selectivescheduling frequency-selectivescheduling ����������������������������������������������������������������������1515

Testingnetwork

Testingnetworkcoverage���������������coverage������������������������������������������������������������������������������������������������������������16�����16

TestingLTE

TestingLTEhandover����������handover�����������������������������������������������������������������������������������������������������������������������16�����16

ValidatingLTE

ValidatingLTEbackhaulbackhaul�������������������������������������������������������������������������������������������������������������������������������18���18

VerifyingLTE

VerifyingLTEhandsetIOThandsetIOT����������������������������������������������������������������������������������������������������������������������������18��18

Validatingdevice

Validatingdeviceconfiguration�����configuration���������������������������������������������������������������������������������������������������������������18��18

Table o Contents

(2)

(3)

Appendix

Appendix: Outline for a Basic Phase 1 L: Outline for a Basic Phase 1 LTE TTE Test Planest Plan� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 2020

Overview������������

Overview�������������������������������������������������������������������������������������������������������������������������������������������20�����20

Achievabledataratesandlatency:single-userthroughputforUL/DLandTCP/UDP��������������20

Achie

Achievabledatavabledataratesandlatency:cellthrougratesandlatency:cellthroughputandMUthroughputforUL/DLandTCP/UDPhputandMUthroughputforUL/DLandTCP/UDP���21���21

Achievabledata

Achievabledataratesandratesandlatency:latencylatency:latency������������������������������������������������������������������������������������������������������2121

Intra-LTE

Intra-LTEmobility:mobilitymobility:mobilityandhandhandoverperformanceandoverperformance�����������������������������������������������������������������������21���21

Achievabledata

Achievabledataratesandratesandlatency:applicationlatency:applicationperformanceperformance����������������������������������������������������������������������2222

Coverageand

Coverageandcapacityradiocapacityradiofeaturesefficiencyfeaturesefficiencyandgainandgainassessment:linkassessment:linkbudgetbudget����������������������������������2222

Coverageandcapacityradiofeaturesefficiencyandgainassessment:scheduler�������������������22

Evaluationof

Evaluationofantennaconfigurationantennaconfigurationoptions��������options�����������������������������������������������������������������������������������23�����23

Self-configurationand

Self-configurationandself-organizingnetworkself-organizingnetworkfeaturesfeatures�������������������������������������������������������������������������23���23

Evaluationof

Evaluationoffrequencyrfrequencyreuse:oneeuse:onedeploymentscenario�����deploymentscenario�����������������������������������������������������������������23������23

BasicQoS:

BasicQoS:useruserdifferentiationbetweendifferentiationbetweennon-GBRnon-GBRuserswithuserswithdifferentQCI��������������differentQCI������������������������������2424

BasicQoS:

BasicQoS:user-differentiationbetweenuser-differentiationbetweenGBRGBRandnon-andnon-GBRuGBRusers���������sers������������������������������������������������24���24

Basicapplicationperformance:webbrowsing,streaming,voicecalls,e-mail,VPN,on-linegaming��25

References

References � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 2626

EPSspecification

EPSspecificationreferencesreferences��������������������������������������������������������������������������������������������������������������������������26��26

3GPPreferences������ 3GPPreferences�������������������������������������������������������������������������������������������������������������������������������������26�����26 NGMNreference������������ NGMNreference����������������������������������������������������������������������������������������������������������������������������28������28 ETSIreference ETSIreference�����������������������������������������������������������������������������������������������������������������������������������������������28�����28 Glossary Glossary � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 2929

(4)

Specifc LTE Test Areas

Overview

Creatingan

Creatinganoverallframeworkforoverallframeworkforthetesting,thetesting,evaluation,andoptimizationofevaluation,andoptimizationofLTEandLTEandSAEisSAEisalargealargeandand

complextopic�This

complextopic�Thisdocumentprovidesadocumentprovidesastartingpointthatstartingpointthatcoversthecoversthetopicattopicatarelativelyhigharelativelyhighlevel�level�

Lookingacrossthe

LookingacrossthelifecycleofalifecycleofatechnologysuchtechnologysuchasLTE/SAE,asLTE/SAE,thetools,thetools,processes,andmeasuresprocesses,andmeasuresmustbemustbe

tailoredtosuitorganizationalprioritiesatspecifictimeswithinthelifecycle(Figure1)�JDSUprovides tailoredtosuitorganizationalprioritiesatspecifictimeswithinthelifecycle(Figure1)�JDSUprovides acost-effectivesetofsolutionsthatenablereusingassetsacrossthelifecycle�Thisensuresnotonly acost-effectivesetofsolutionsthatenablereusingassetsacrossthelifecycle�Thisensuresnotonly completecoveragebutalsothereuseofresultsandassets,leadingtoasolidreturnoninvestment(ROI)� completecoveragebutalsothereuseofresultsandassets,leadingtoasolidreturnoninvestment(ROI)� TheexampleslistedbelowaddressdifferentaspectsoftestingatvariousstagesintherolloutofLTE/ TheexampleslistedbelowaddressdifferentaspectsoftestingatvariousstagesintherolloutofLTE/ SAE�TheseareofferedasanoverviewofthemajorelementsoftheLTEdeploymentlifecycle� SAE�TheseareofferedasanoverviewofthemajorelementsoftheLTEdeploymentlifecycle� •

• TTechnoechnologyellogyeldtrialsdtrials

−

− EvaluatethetechnologyagainstNGMNorEvaluatethetechnologyagainstNGMNorotherindustryrequiremenotherindustryrequirementsts

− Testandvalidatetechnologyimplementations

− Adoptaprocessthatismoreopenthanfullyclosedbilateraleldtrials

•

• LabLabtestestintingg

− EvaluateeNodeBschedulingperformance

− Evaluatesecurityandbillingpolicies

−

− EvaluateserviceperformanceincontrolledenvirEvaluateserviceperformanceincontrolledenvironmentsonments

− EvaluateUEperformance

− EvaluateMIMOperformancegains

•

• FieldtFieldtrialsarialsandvendvendorndorevaluaevaluationtion

−

− Evaluateend-to-end(E2E)Evaluateend-to-end(E2E)performanceperformance

−

− EvaluatenetworkEvaluatenetworkcoveragecoverage

−

− EvaluatecellandnodeperformanceunderEvaluatecellandnodeperformanceunderloadedconditionsloadedconditions

− Evaluateself-optimizingnetworkcapabilities

−

− EvaluateIRAEvaluateIRATcapabilitiesandperformanceTcapabilitiesandperformance

−

− PerformKPImonitoPerformKPImonitoringandbenchmarkingringandbenchmarking Figure 1. A simple model

Figure 1. A simple model of the LTE deployment lifecycleof the LTE deployment lifecycle

Techno

Technologylogy

eld trials

eld trials Lab trialsLab trials

Field trials Field trials and and vendor vendor evaluation evaluation Friendly Friendly customer customer trials trials Commercial Commercial launch launch Optimzation Optimzation and and wider wider deployment deployment

(5)

• Friendlycustomertrials − Understandend-userQoE/QoS − Conrmhistorictroubleshootingcapabilities − TesthandsetIOT,conformanceandpre-conformance − Monitorhandsetperformance − Includetrending,statisticsandVIPreporting • Commerciallaunch − Performservice-andaccesstechnology-awaremonitoring − VerifyE2Evisibilityandtroubleshootingcapabilities − Validateandunderstandsubscriberbehavior − UnderstandoverallnetworkandIRAThandover(HO)performance − Includeintegrationwithnode-loggingcapabilities • Optimizationandwiderdeploymentphase − ValidateandensureSONcapabilitiesareworking − Locateareasforexpansion − Monitorbackhaulperformanceimpactonend-userQoE − Benchmarkserviceperformancebetweenmacroandfemtoroll-out Theremainderofthisdocumentfocusesonthe“fieldtrials”portionoftheprocess�Eachsection describesthetestcasesmostoperatorswillwanttoundertakeandalsooutlinesthetoolsthatwillensure effectiveperformanceofsuchtests�Itisassumedthatnotalloperatorswillperformallstepssowillneed tomatchthetestcases(andpossiblesolutions)touniqueneeds�

Gaining deeper insights

Foryears,personnelfromacrossAgilentTechnologieshavebeen involvedwithtechnologystandardizationandtestdevelopmentfor LTEandSAE�Since2008,Agilenthasactivelyprovidedtoolstohelp developerstakeLTE/SAEforwardtothemarketplace�InApril2009,the companypublishedacomprehensivebookcalledLTEandtheEvolution to4GWireless:DesignandMeasurementChallenges(ISBN:978-0-470-68261-6)�Thisresourcehasreceivedsignificantandwidespreadpositive feedbackfromthewirelessindustry�InMay2010,JDSUacquiredthe NetworkSolutionsDivisionfromAgilent�Withthisacquisition,several contributingauthorstothisLTEbook—includingtheauthorofthis applicationnote—transitionedfromAgilenttoJDSU�Thebookis justoneexampleofhowAgilent—andnowJDSU—contributetothe overalllandscapeoftheLTEandSAEindustry�Asacompaniontothebook,thisapplicationnote containsrelevantreferencestotheLTEbook,whichcanbestudiedseparatelyinpursuitofadeeper understandingofthevarioustopicsandconceptsinvolvedinLTEtesting�

(6)

Perormance testing WithLTE,muchofthefocushasbeenonincreasingsystemperformanceinareassuchasend-user throughput,latency,andidle-to-activetransitions�Althoughtheseimprovements—alongwith significantlyimpressivesingle-userdatarates—areimportantformarketingreasons,inthemselves theyplayalimitedroleinhowrealuserswillutilizeadeployedLTEnetwork�Thus,measurementsare neededforbothsingle-userpeakratesandmulti-userthroughput�Inanalyzingmulti-userthroughput andoverallcellcapacity,itisimportanttounderstandtheimpactofacell’srealdistributionofusersas wellasthemobilityandusagepatternsofthoseusers�

Single-user throughput testing

Intheory,single-userthroughputtestingisquitesimple�However,inrealityitcanprovetobequitetricky� Asanexample,thefirstaspectstounderstandare what to measure and how to benchmark the result � LookingatthepublicresultsoftheLTESAETrialInitiative(LSTI)ProofofConcept(PoC)group,it canbeseenthatpeakrateswillvaryfromafewhundredkilobitspersecondatthecelledgetoover 150Mbpsinverygoodradioconditions(forexample,ina20MHz2x2MIMOsystem)�Inpractice, measuringthiswiderangeofratescouldbeperformedwithadedicatedhardwaretrafficgenerator thatwillhaveguaranteedperformanceandwillproducetrafficpreciselyaccordingtoananticipated trafficprofile�Suchaprofilecouldincludevaryingdatarates,differenttypesoftraffic,anddelay characteristicssuchasjitter�Otheralternativesincludesoftware-basedtrafficgenerators(for example,iperf)orsimplygeneratingthetrafficfromexistingstandardapplicationserversusedfor FTPorvideostreaming� Ifasoftware-basedtrafficgeneratorisused,itisimportanttounderstandtheconditionsunderwhich itwilldeliverreliableresults�Ingeneral,currentlyavailablesoftware-basedtrafficgeneratorsproduce acorrectaveragethroughput;however,theinstantaneousvariationscouldbesignificant�Togenerate highrates,dedicatedCPUresourcesmustbeconsistentlyavailabletothegenerator�Shouldother processesstartonthetrafficPC,andthisisoftenthecasewithcertainoperatingsystems,theremay becomplicatedsideeffectsintheEPCortheeUTRANbecausebufferingmightoccurunexpectedly� Consequently,theremaybeagapinthegeneratedtrafficandthusnothingtotransferovertheair interfaceforagivenTTI� Ifnotmanagedproperly,theuseofanexistingapplicationserver(forexample,anFTPserver)to generatethetrafficcouldalsoproduceunexpectedsideeffects�Forexample,anFTPservernormally accessesafilefromatraditionalharddisk�Ifmultipleuserstrytodownloadthesamefileatthesame time,therecouldbeabottleneckcausedbytheFTPserverratherthantheLTEairinterface(ingood radioconditions)�ItispossibletosetupanFTPservertomanagethisifpropercareistakenandthe appropriatedisksystemisapplied;however,oversightsassimpleasanimproperlyconfigureddiskcan skewtheoverallresults� Whenitcomestomeasuringtheperformance,oneessentialruleofthumbistounderstandthebasic aspectsthatwillcontroltheresults�Forexample,anunderstandingoftheunderlyingradiolayer transportsettingsiscrucial�HARQprovidesanexample:ifHARQisnotenabled,thelinkwillactually deliverahigherthroughputingoodradioconditionscomparedtotimeswhenHARQisnotswitched on�Ontheotherhand,reliabletransportdependsonHARQbeingswitchedon�Ifnot,problemssuch asasignificantamountofTCPretransmissionwillcropup,leadingtoaverypooreffectivethroughput� EvenifHARQisswitchedon,theamountofHARQretransmissioncanbeconfigured�Settingthisto averylowvaluewillincreasetheinstantaneousthroughputbutwillleadtoapooreffectivethroughput undernon-idealradioconditions�

(7)

Foradetailedunderstandingofbearerthroughput,measurementsshouldbemadeatdifferentlayers intheprotocolstack—MAClayer,IPlayerandUDPorTCPlayers—afterapplicableretransmissions� Duringthesemeasurements,itisimportanttounderstandandrecordtheactualsettingsthatwere configuredfortheMAClayerandtheTCPlayer,andifanyspecificserviceorapplicationlayersettings havebeenapplied(forexample,iftheapplicationusingTCPhadoneormultipleTCPflows)� InsomeveryspecificcasesitmaybeusefultomeasurethehighestpossiblebitratethattheLTEair interfacecandeliver�Forthismeasurement,therecommendedapproachistomeasurethelossless (forexample,zeropacket-lossthroughput)ofasustainedUDPstream�Insuchcases,thespecific MAClayersettingsshouldberecordedaswellastheUDPpacketsize�Itcouldaddsubstantialvalueto performanRFC2544testovertheLTEconnection;thiswillstepthroughseveraldifferentpacketsizes andthroughputratestolocatethelosslessthroughputforeachrelevantpacketsize� Tounderstandwhythroughputischangingindifferentenvironmentsatdifferenttimes,thebest approachistorecordasetofLTE-relevantparameterswhileperformingthethroughputtest�These parametersshouldincludetheinputstotheeNodeBschedulingdecisions(forexample,CQIforall oftheranks(widebandandsub-band),thePMI,andtherankindicator)andinformationaboutthe resultingeNodeBschedulingdecisions(forexample,selectedmodulationandcoding,MIMOmode, etc�)�Whentestinguplinkperformance,therelevantpower-controlinformationshouldalsobelogged� JDSUprovidesallofthetoolsandprocessesneededtoplan,perform,evaluateresults,andprovide reportsforsingle-andmulti-userthroughputtesting�ThisincludesbothtoolstogenerateE2Etraffic aswellastoolstologdatafromhandsetsandrelevantnetworkinterfaces�Fortestinginthefield,signal sourcesavailablefromothervendorsemulateuplinkinterferenceandloadingcausedbyusersinother cells�Becauseloadinghasasignificantimpactonthroughputratesitshouldbeincludedaspartofany realisticevaluation�ThegeneratedinterferenceshouldideallyberepresentativeofanLTEairinterface inbothULandDL�AlthoughwhitenoisewasacceptableforUMTS,whichusesanoise-likeCDMAair interface,itdoesnotrepresentthepowervariationsacrossfrequencyandtimethatcharacterizeLTE’s OFDMandSC-FDMAmodulationschemes�Rapidvariationsmaytrickschedulersintomakingeither optimisticorpessimisticpredictionsofthechannelconditionsandsubsequentmodulationandcoding schemethatcanbereliablytransmitted�JDSUtoolsprovidethemeanstoevaluatethisimpactbefore networksbecomeheavilyloaded�

Cell performance verification with multi-user throughput testing

MeasuringtheperformanceofanLTEsysteminasingle-usercaseprovidesabasicunderstanding uponwhichamorerealisticanalysisofthemulti-usercasecanbeperformed�Itshouldbe understoodthat,inmanycases,single-userpeakthroughputperformancewillbemuchhigherthan eventheaggregatecellthroughputwithseveralusersactivelydownloading�Fromindustrystudies performedbyLSTIandothers,DLcellthroughputisexpectedtobearound35-40Mbps(assuming 20MHzwith2x2MIMO)with10usersspreadoverthecell,allwithfullbufferdownloadsanda proportionalfairscheduler�1,2 1�eseexpectationsarerelativelyconsistentwithprevious3GPPsimulations� 2�eterm“fullbuer”referstotheideathattheeNodeBisconstantlyschedulingdatafromtheS1interface�ebuermustalwayscontain enoughdatatofullysaturatethelinkeveninthecaseofaperfectRFenvironmentandthehighestthroughoutrate�

(8)

Whenbasicmulti-usertestingisperformed,theresultsmayvarygreatly,spanningfromvalues belowtheanticipatednumberstoratesnearthesingle-userpeakperformance�Thereasonforthis widevariationisduetotwokeyfactors:theoverallbehavioroftheeNodeB’sschedulingandthe environmentforeachindividualuser�Itisthereforeimportanttosetupthetestinarepeatablewayto ensurethatresultsareusefulontheirownandcanbecomparedagainstsimilartestsineitheradjacent cellsinthesamenetworkorwithcellsthatusenetworkequipmentfromanothervendor� Theprocessofsettingupatesttoverifycellperformancewithmultipleusershastoomanystepsto presenthereindetail�Inoutlineform,herearethekeysteps: 1� Evaluatethedistributionofsignalqualityinthenetworkthroughexhaustivedriveorwalktesting tobuildaCDF� a� IftheCDFsfordifferentcellsaresignificantlydifferent,theendresultwillbesignificantlydifferent� b� Thedistributionwilldependonthetypeofterraincoveredandwherethewalkordrivetestis performed�Forexample,includingindoorlocationswillproducelowersignalqualitiesthan outdoor-onlydrivetesting� c� Itisexpectedthatroughly80percentoftheLTEdatatrafficwillbegeneratedindoors� 2�DistributetheUEsaccordingtothesignalqualityCDFtoobtainapropertracprole� a� ThechoiceofdistributionshouldmatchnotonlythemeasuredCDFbutalsotheanticipated distributionofrealusersintherealnetwork�Forexample,ifitisanticipatedthatseveralusersare locatedinagroupinapubliclocation(forexample,anairportorcafé)thenitisadvisabletoplace someoftheusersinsimilarpatterns� b� DifferentMIMOconditionsmightbestressedduetothistypeofdistribution(forexample ,multi-usergainsfromMIMO)� 3�Intherealnetwork,locatetheusersaccordingtotheplanneddistribution� a� Notethatitwillbenearlyimpossibletogetanexactmatchsoitisimportanttoinsteadlocatethe usersinanareasimilartotheoneidentified�Makesurethattheactualradioconditionsofthe chosenlocationsareloggedandstoredforthecompletedurationofthetest� b� Forcasesinwhichsomeoftheuserswillbeinamovingenvironment,makesurethatthiscanbe managedinacontrolledway� 4�EnsureOCNGisenabledintheDLfortheadjacentcellstocreateDLloadinginthecell� 5�EnsurerealisticULloadingisgeneratedfromUEsinadjacentcells� 6�GeneratetractothetestUEsandensurethateachUEissetuptoreceiveortransmittracwith fullbuers� 7�Logalloftherelevantparametersasidentiedearlierincludingthelocation(latitudeand longitude)ofeachUE� a� Logallrelevanttrafficfromthenetworkinterfacestoensurethatfullbufferswerepresentforall UEsinallconditions� b� VerifythatallUEswereactiveandthattheindividualUEthroughputratesarerealisticcompared toexpectations� 8�CorrelatetheUE’sentireindividualthroughputforMAC,IP,andUDPorTCPlayers� a� IfoneorseveraloftheUEsweremoving,theaggregatedthroughputmaychangeovertime dependingontheschedulingandspecificradioperformanceofeachdevice’senvironment�

(9)

Itshouldbenotedthatthisisnotarealisticwaytotestacell’sactualperformanceinarealnetwork scenariowhenbuffersarenotalwaysfull�Instead,itisatestofhowtheeNodeBwouldbehavein averyspecificscenario�Thiscouldnormallybeconsideredaworst-casescenarioforaspecific distributionofusers�

Realistic multi-user throughput testing

Despitethecomplexity,therearestilllimitationsinmulti-usertestingthatusesfullbufferdownloads anduploads�Inarealistictrafficscenario,usersareexpectedtohaveburstytrafficprofiles,possibly comprisingHTTP,voice,FTP,IM,e-mail,etc�Inthefull-buffercase,userswithgoodconnections downloadmoredatathanuserswithlowdatarates,whichbiasesresults�Althoughthisispartlytrue, userstendtodomorewithafasterconnection—thefull-buffercaseisperhapstooextreme�The question,then,iswhatwouldbeamorerealistictrafficmodel?Woulditbevaluabletopursuethis testingand,ifso,whatistheaddedvalue?Theanswerdependsonwhichphaseofthetechnologythe specificoperatorisinatthetime�Iftheoperatorisabouttoperformvendorselection,thiscouldprove tobeanessentialtesttoensurethattheequipmentbehavesappropriatelyintermsofscheduling,ability todeliverthedesiredQoS,andoverallfairnesstothedifferentusers� ItisalsoimportanttounderstandthatdeliveringtheexpectedQoSorbeingabletoprovidefairness inthesystemisnotnecessarilyadifficulttaskforaneNodeB�Inpractice,thedifficultyistheabilityto delivertheexpectedQoSwithaminimumofoverheadfactorsthatimpacttheoverallcellcapacity� Thefollowingisaproposedtesttohighlightcertainpossibledeficienciesinasystem� 1�QoSoverheadprovisioninganalysis a� Setupastaticmulti-usercelldownloadwith6to10usersperformingfull-bufferdownloads� b� Measuretheoverallcellcapacity� c� Reassignoneuserasamobileuserwithafixed-rate UDPstreamwiththesamemegabit-per-secondratethatwasachievedinthefull-bufferdownload� i� SettheQoSparametersforthisusertomatchthefixedrateoftheUDPstream� ii� Verifythatthethroughputrateisintherangeof2to5Mbps� d� Measuretheoverallcellcapacity�

e� Movetheuserfrommediumtopoorradioconditioninfivestepsandrepeatthecell-cap acity measurementsforeachstep� f� Movetheusertothecelledgeforthespecificdataratethatwasprovisioned� g� Measuretheoverallcellcapacity� ComparethecellcapacityrstwithandthenwithouttheQoSprovisionedforthebandwidththat canbedeliveredwithoutanyactualimpactontheoverallcellcapacity�encompare,stepbystep, theoverallimpactofusermobilityoncellcapacityandguaranteedQoS� 2�Testtheimpactofrealistictracbehaviorontheoverallscheduling a� Setupagroupofusers(6to10)witharealistictrafficprofileforthetypeofusersenvisionedfor theLTEnetwork�Thefollowingareexamplesofusescenarios� i� Downloadfivewebpageswitha30-seconddelaybetweeneachdownload�Eachpage contains20objectsforatotalof1�4MBperpage� ii� Receiveane-mailmessagewithalarge(forexample,10MB)attachment�

(10)

iii� Send10e-mailmessages,fourwith2-MBattachmentsandsixsmalloneswithonly50KBof dataineach� iv� ListentoInternetradio(256Kbps)� v� PerformabackgroundFTPdownloadofsevenlargefiles(totalof100MB,similartoa monthlyoperatingsystemupdate)� vi� MakeoneVoIPcallusing12�2KbpsonaguaranteedQoSbearer� Notethatthetestconditionsmustbesetupsuchthattheradiointerfacewillbecongestedin certainconditionsduringtestexecution�Usethepreviouslymeasuredanticipatedcellcapacity numberstoselectthecorrecttest-caseparameters� b� Thesequenceofeventsoccursinparallelforeachuser;however,theusersareseparatedintime by10secondseach� c� EachuserwouldbelocatedinaspecificradioenvironmentdefinedbytheCDFforthespecificcell� d� LetthetestruntocompletionwhileloggingalldatafromboththetraceUEsandthenetwork basedprobes�Ifpossible,alsologthedatafromtheLTEUuinterface� Thedatashouldthenbeanalyzedtorevealactualbehaviorduringcongestedconditions�One shouldspecificallystudytheimpactofthescheduleronboththeguaranteedQoStraffic(VoIP) andthestaticRTPstreamfortheInternetradiobecausethesetwoservicesarethemostlikely todegrade� Asimpleandbasicmeasureistobenchmarkthetotaltimeofcompletionforthetestsequence (excludingtheInternetradiostreamandtheVoIPsession,whichcouldrunindefinitely)�The minimumcompletiontimeshouldberelativelyeasytocalculatebasedonthepreviously measuredperformanceforthemulti-userandfull-bufferdownloadscenarios�Thedifference fromthisminimumtimeshouldbeanalyzedtounderstandtheoverallefficiencyofthesystem� Addingmobilitytothistestcasewouldallowforfurtheranalysis�However,itwouldprobablyadd somuchcomplexitythatitcouldnotbejustifiedasabasiccase� Allaspectsoftheschedulingandperformanceoftheradioenvironmentshouldbeanalyzedto understandtheefficiencyineachlayerandduringeachprocess�Thiswillalsohelprevealwhich conditionsleadtosuccessfulschedulingresultsandwhichcauseschedulingissues� JDSUcanprovidethetoolsandtheprocessestoexecuteandanalyzetheresultsfromthesetypesof tests�JDSUtoolscanhelpcharacterizethebehaviorandallowforbothcompetitivebenchmarkingand regressiontestingwhena“laundrylist”mustbemaintainedforaspecificsupplier�

Idle-to-active transition times

OneofthemainexpectationsofLTEistoprovidean“alwaysconnected”experienceforendusers�This isachievedinpartbyensuringaswifttransitionbetweentheidleandactivemodes�TheoverallRRC statemachinehasbeenoptimizedandthenumberofpossiblestateshasbeenminimizedtoensure reducedcomplexity,lowerpowerconsumption,andfastertransitiontimes� Tomeasuretheidle-to-activetransitiontimesonemustbeabletoeitherfullycontrolUEbehavior orbeabletologalloftheassociatedsignali ngtoensurethedataisavailabletobemeasuredfromthe overalltraffic� JDSUtoolscanmeasureidle-to-activetimeaswellasotherrelevanttransactionandproceduraltimes� Thesemeasurementscanbeperformedusingeitherdatafromatracemobilealoneordatacombined fromtheLTEandEPCnetworklinkstoenabletruecorrelatedE2Emeasurements�

(11)

Latency testing

Theoverallexperienceismorethanthespecificbandwidththatanendusercanreceiveandhow quicklythenetworkwillenableswitchingbetweentheidleandactivemodes�Theexperiencealso dependsontheE2ElatencyofapackettransitioningthroughthecompleteEPCandeUTRAN� KPI veriication and calculation

TheJDSUbookLTE and the Evolution to 4G Wireless: Design and Measurement Challengesincludes anextensivesectionaboutKPIs,thecalculationmethods,andtheoverallmethodology(pleasereferto Chapter6)�Asaresult,theKPImaterialcoveredhereiskepttoaminimum� Itisunfortunatethatthephrases“keyperformanceindicator”and“KPI”havebecomecommonly misunderstoodandmisused�Atthemostbasiclevel,aKPIisnothingmorethanastatisticora measurement�However,itisthetestobjectiveormarketrequirementforagivenservicethatallowsa particularstatisticormeasurementtobeconsideredasakeyindicatorofperformance� Eventheterm“performance”canmeanverydifferentthingsdependingonthetestingcontext�For example,serviceperformanceforVoIPmaybemeasuredintermsofjitter,latency,anddropped packets�NetworkperformancemaybemeasuredbythenumberconcurrentVoIPusersthatcanbe servedwithanacceptablelevelofjitter,latency,andpacketloss�Thus,whenmeasuringqualityor performance,oneofthekeychallengesisagreeingondefinitionsthatenableconsistentinterpretations ofresults� Accordingto3GPP,KPIsgenerallyfitintooneoffivecategories:accessibility,retainability, integrity,availability,andmobility�Thelistissometimesexpandedtoincludethecategoriesof utilizationandusability� 3GPPKPIstandardizationeffortsarelargelyfocusedonmeasurementsrelatedtotheenduser’s perceivedQoS�Thesemetricstendtobemoreoperator-centricastheyrelatespecificallytomeasuring theabilityofcustomerstoobtainandmaintainaconnectiontothenetworkandtherebymakeuseof oneormoreavailableservices� KPIsarebestunderstoodinthecontextoftheactualobjectiveofthemeasurement�Eachpartofthe networkhasdifferentresponsibilitiesassociatedwithdeliveringasingleservice�Therefore,LTE-specificKPIsfocusontheeUTRANitselfand,inmanycases,relyontheeNodeBtoactuallymeasure itsownperformance�OnechallengeforanNEMisdefiningawaytoverifythatKPIscalculatedbythe eNodeBarecorrect,especiallywhenrunningathighloadorfullcapacity�Anotherchallengeforboth NEMsandWSPsismakingtheshiftfromsimplylookingatKPIstotroubleshootingandidentifying therootcausesofproblems� ForeachKPIcategory,eachservicemayhaveadifferentQoSprofileorQCIlabel�Toidentifythe performanceofeachtypeofservicebeingaccessed,themeasurementsshouldbemadeonaper-QCI basis�AdditionalKPIsineachofthesecategoriesshouldbeconsideredinordertoevaluatetheend-to-endusabilityandmanageabilityofaservice� Keyaspectsofawell-designedend-to-endtestsystemarethedatasourcesandthepossiblemonitoring pointsthatexistinanLTEandSAEnetwork�Someofthetopicsdiscussedherearealreadypartofthe industrystandards�Otherareasmayormaynotbepartoffuturestandardizationeffortsby3GPP,ETSI, orotherindustrybodies�

(12)

Fundamentaltothetopicofdatasourcesandmonitoringpointsisanunderstandingofmeasurement theory,basicphysicallaws,andhowtheLTE/SAEtechnologyworksandisdeployed�Considerthe followingexamplecommontobothLTE/SAEandUMTSinwhichtwoengineersneedtomeasurethe RRCconnectionsuccessratiointhenetwork� • erstengineer,whoisexperiencedindrivetesting,commissionsatargeteddrive-testcampaign, measuringtheRRCconnectionsetupsuccessratioforawidearea�Clearly,thenumberof measurementpointsisdirectlyrelatedtothedurationofthetestingandthenumberofactual attemptsperunitoftime�Forthisexample,theengineerconcludedthattheRRCconnection successratiowas98�5percent� • esecondengineerisaccustomedtonetworkcountersandlink-monitoringtools�Asaresult, heextractslogsfromthesystem�isprovideshimwithareportofalloftheRRCconnection attemptsforthefullnetwork;hismeasuredRRCconnectionsuccessratiois99�5percent� Whyisthereadifferenceofnearlyonefullpercentagepoint?Theanswerisfundamentaltotherestof thisdiscussion� Thedifferenceintheresultsisnotduetoflawsinthedatasource,beitthedrivetest,thenetwork counters,orthelink-monitoringtools�Thedisparityiscausedbythetwoengineersmeasuringdifferent networkproceduresfromdifferentangles� • edrive-testmethodanalyzesnetworkperformanceasseenfromasinglehandsetatspecic physicalpointsinthenetworkatdistinctpointsintime� • enetworkcountersandlink-monitoringtoolsrecordallofthetracandalloftheoccurrences ofsignalingandusertractheyaredesignedtomonitor�isisadierentframeofreference:it analyzesnetworkperformanceasseenonthenetworkandatthenetworkmonitoringpoints� Differencesbegintoaccumulateifanydrive-testlocationsarewelloutsidetheactual,andpotentially, intendednetworkcoveragearea�Asaresult,RRCconnectionrequestsmadeoutsideofthenetwork coverageareawillberecordedbythedrive-testsystembutnotbythenetworkcountersorlink-monitoringsolutions�Thishighlightsakeypoint:anextensivedrive-testcampaignprovidesadditional informationbeyondwhatnetworkcountersorlink-monitoringtoolscanprovide� Intheoptimizationcommunity,itisgenerallyagreedthatKPIsshouldbecomparedtoeachotheronly whentheyarederivedfromthesamedatasourceorwhentheyarenormalizedtoremoveanybiasdue tomethodorsource�Thisisespeciallytrueifcomparisonsshowunexpectedresults�Today,thelackof propercomparisonsisoneofthelargestcontributorstounsoundoptimizationdecisionsinthemobile industry�Validcomparisonsandmeaningfuloptimizationcanbeensuredonlyifastringentand coherentagreementondatasourcesandmonitoringpointshasbeensettledinadvance� 3 WhetherengagedintheR&Dprocessortheoptimizationphase,onemustoftenchoosebetween severaldifferentstrategieswhendevelopinganLTEtestplan�Selectingthemostcost-effectiveand results-effectivestrategyisoneofthemostimportantdecisionstobemadeearlyineachphaseofthe work�Onceastrategyisselected,theboundaryconditionsofitsapplicabilitymustbeestablished�Itis importanttonotethatastrategythatisappropriateforonephaseoftheworkprobablyhassignificant shortcomingsinanyotherphase�Inotherwords,itisseldomagoodideatousethesamefundamental KPIsbecausethedatasourceswillprovidedifferentresultsindifferentphasesofanetwork’s deploymentandmaintenance� 3�Pleasenotethatthisissimplyamatteroffundamentalmeasurementtheory�Itisnotintendedtobeadiscussionaboutthepotentialriskof resultsbeingincorrectduetoacertainmeasurementtoolnotworkingasdened�

(13)

Toillustratethisidea,consideracaseinwhichQoSiscontrastedwithQoE�Monitoringtheend-user IPtrafficonamobilenetworkprovidesafullanddetailedunderstandingofthetrafficflows(TCPor UDP),theapplications(voice,video,HTTP,e-mail,etc�),andpotentiallyperformance�Somebelieve thatthemonitoringofonlytheUDPorTCPflowswillprovideenoughinformationabouttheend-user QoStobeabletodeduceagoodapproximationoftheend-userQoE�Asthefollowingscenarioshows, thisisnotthecaseforapplicationssuchasstreamingvideo� • Auseriswatchingstreamingvideoonhishandset,buttheradioqualityisnotsucienttodeliver thefullbandwidthovertheairinterface�WhentheUDPstreamismeasuredinthecorenetwork, nodegradationoftheRTP/UDPstreamisobserved�reeotherlocationsprovideabetterplaceto observethedegradation:ontheairinterface,ontheuser’shandset,orbetweenthetwoendpoints oftheRLCentity(intheUTRANbetweentheUEandtheRNCorintheLTEeUTRANbetween theUEandtheeNodeB)�enetworkmonitoringtoolinthecorewouldreportahighandstable bitrate(goodQoS);however,theenduserwouldreportapoorQoE� • Shiingthescenarios,assumethattheenduser’sapplicationisquality-awareand,duetotheradio conditions,signalsthatthevideostreamingservershouldchangethebitrateofitscodec�When thisnewRTP/UDPbitstreamappearsonthenetwork,thenetworkmonitoringtoolwillassociate thechangewithalowerQoSstreambecauseithasalowerbitrate�Ontheotherhand,theradio conditionsaregoodenoughtodeliverthisadaptedbitstream,andtheenduserQoEhasnowincreased� ThisscenarioshowsthatthecrucialelementintheQoEisnotthebitratemeasuredinthecorenetwork, butrathertheend-to-endabilitytodeliveraspecificservicetotheenduser�Theapplicationdomain will,inthiscase,ensurethatthebestpossibleQoEisachieved�Therefore,themonitoringtoolmustbe application-awaretodelivercorrectQoSmeasurementsthatleadtothecorrectestimationofend-userQoE� Validating LTE voice

Oneoftheto-be-defineditemsfortheEPSishowcircuit-switchedservicessuchasvoice,CSUDIvideo, SMS,LCS,andUSSDwillbemanaged�Fouralternativesarecommonlyconsidered� • Circuit-switchedfallback(CSFB) • VoiceoverLTEgenericaccess(VoLGA) • VoiceoverIMS(VoIMS) • Proprietaryoptions CSFBandVoLGAarebothstandardizedandcouldbereadilyimplemented�VoIMSislikelytofollow andisanticipatedtobeawidespread,long-termsolution�Amongthese,CSfallbackinEPSisdescribed indetailbelow;theotheroptionsarecoveredinbrief� Anotheraspectthatisnotyetspecifiedin3GPPR8istheuseofavoicecodec�Severaldifferentoptions areathand;however,duetoalackofagreement,thispartofthestandardizationmightbedelayeduntil R10�Onekeyreasonforthedelayisthelackofclarityontheobjective:shouldthequalitybeimproved orshouldthecapacitybeimprovedbythechoiceofcodec?Itislikelythatachoiceofcodecforinitial EPSdeploymentswillbebasedonamutualbilateralagreementbetweentheUEvendors,theoperators, andtheEPSprovider�ItwouldnotbeasurpriseifAMRandAMR-WBwereusedinitially� ItiscrucialtounderstandthattheITUhasalreadymovedaheadwiththedefinitionoftheG718codec� G�718isbuiltontheAMR-WBfoundationand,forthemostpart,providesthequalityofAMR-WB at12�65KbpsonthesamecapacityasAMR7�95Kbps�Overall,thismeansthatG�718providesa57 percentincreaseincapacitywithaverylimitedimpactonspeechdelay�Adetaileddescriptionis outsidethescopeofthispapersothereaderisencouragedtostudyITU-TRec�G�718infulltofurther understandthistopic�