Lte Operations and Maintenance

(1)

LTE OPERATIONS AND MAINTENANCE

(2)

INTRODUCTION

• LTE Should Revolutionise O&M in Mobile Networks

• Self Organising Networks (SON) Specified in 3GPP REL 8

36.902 • In Reality Early SON implementations have proved

un-Predictable (ANR) or non trivial to manage (eNB Autconfig)

Predictable (ANR) or non trivial to manage (eNB Autconfig)

• SON will become more reliable and understood and More

SON Features will be released Step by Step

• Currently Most LTE O&M operations are using familiar apps

from 2/3G

(4)

• No Object Browser In APPLAU (Use TLUI Status or SiteBrowser) • iOMS located in NetAct Rack . Responsibility of OSS Engineer? • Many New Parameters and Counters particular to the technology • As no RNC in middle it may be harder to trace Transmission IP

issues eTe

MAIN DIFFERENCES IN LTE O&M TASKS FROM WCDMA

4

issues eTe

• As relatively New Technology Must be aware more device compliance issues

• Licence Management ( Use of SWAM)

• Challenges to O&M to understand SON automated tasks eg auto ANR ,PCI etc

• Most important to Maintain Voice cabability (CSFB,SRVCC) • Greater Mobility Issues to Consider LTE->3G->GSM

(5)

OAM Architecture

From the users’ point of view, managing the Nokia Networks Mobile LTE/EPC network is very similar to that of Nokia WCDMA when

NetAct is used.

Management system traffic to and from LTE/EPC network elements always goes through NetAct.

key tool for centralized SON

through NetAct.

The interface between the Flexi Multiradio Base Station and NetAct (iOMS) is based on the BTSOM protocol. It carries all the

necessary data and commands (for example, alarms, measurements, configuration and new software data) to control the network element behavior remotely.

(6)

iOMS in NetAct site

• The LTE iOMS is a part of the NetAct concept. Although

iOMS is running in its own blade unit, it can be located in the same rack as the NetAct Blade System

6

(7)

iOMS SW management iOMS EMs

• iOMS SW can be installed either by

• Using FEWS • Using USB stick

• iOMS has only following EMs supported:

• Application Launcher (Java): • Application Launcher (Java):

• FM GUI

• Active Sessions

• Parameter Tool

• PUTTY

(8)

A.

STATUS MONITORING

B. ALARM MONITORING

C. PARAMATER OPTIMISATION

D. SW ROLLOUT/ROLLBACK

O&M TASKS

8

D. SW ROLLOUT/ROLLBACK

E. CONFIGURATION BACK UP

F. LICENCE MANAGEMENT

G.PERFORMANCE MONITORING

H. TROUBLESHOOTING

(9)

BTS Site Manager Application

• The content in the Task view changes according to the selection in the View bar Task view • Example: BTS Hardware task view consists of Equipment view (shows the BTS cabinet with units/modules and cells) and Properties Fault view View bar Equipment view Properties view

Alarms from the eNodeB and its transport module

(10)

BTS Site Manager Fault Monitoring

Fault view shows the list of faults Faults can be

•

Warning

•

Minor

•

Major

BTS Site Manager shows the faults coming from the site

All

10

•

Major

•

Critical

Fault descriptions

available in online Help

All Active Alarms Shows/Hides Fault history of current manager session Opens Block toggling faults Additional info about the selected fault Show Fault source in EM equipment view

(11)

BTS Site Manager Fault Monitoring

(12)

Introduction

Trial and Rollout Activities

Provides access to multiple predefined BTS sites simultaneously Speed-ups and helps BTS Site management especially

In trials and rollouts

There is no NetAct available

New BTS Site SW version is not supported by NetAct

Uses standard BTS Site Manager interface towards BTS

BTS Site Browser

12

1st _Mile _Core _{Controller/Gateway Sites}

Trial and Rollout Activities

BTS O&M SW version not supported by NetAct

NetAct not in place

(13)

Deployment

• Installed with BTS Site Manager

– BTS Site Browser is installed to BTS Site Manager installation folder

Default location

•Windows: C:\Program Files\Nokia Siemens Networks\Managers\BTS Site\BTS Site Manager\

•Linux: opt\Nokia Siemens Networks\Managers\BTS Site\BTS Site Manager\

Can be started with BTSSiteBrowser.bat or BTSSiteBrowser.sh from BTS Site Manager installation root folder

• Define properties

– You have to create text file where is listed all IP addresses of BTS Sites managed by BTS Site Browser

– File -> Properties

Define text file where IP addresses are listed

Define installation location of BTS Site Manager, so it can be started from BTS Site Browser

(14)

BTS Site Browser main view

List of predefined BTS sites Information about

selected site

14

scanned BTS sites

Block/unblock cells. More info…

(15)

(16)

FAULT MANAGEMENT

16

(17)

Fault Management Architecture NWI3 Interface NWI3 Interface eNBs/iOMS alarms Alarm history Active alarms

(18)

Fault Management

NetAct Monitor application

Alarm Details Alarm List View Explorer • All relevant Alarm Monitoring tools within a Common Desktop. • Possibility to •Alarm List •Alarm History •Alarm Details •Warning List •Rule Editor •Rule Explorer •Object Explorer

•Managed Object List

•Object Search

•Properties

18

Network View Object Explorer • Possibility to launch all OSS5.1 applications and element managers from the desktop •Properties •Filter Explorer •Filter Editor •Quick Search •Network View

•Network View Editor

•Notes

•Trouble Ticket List

•Trouble Ticket Details

Properties and Notes

(19)

FM IOMS GUI

19

(20)

NetAct Application (NetAct OSS5.3 CD Set 1)

20

(21)

Monitoring

• All relevant Alarm Monitoring tools within a Common Desktop

• Possibility to launch all OSS5.x applications and element managers from the desktop

(22)

Alarm Monitoring

22

(23)

Alarm Monitoring Application Alarm Active Alarm List grouped by Details Access to Alarm Manual grouped by Alarm class Alarm Blinks when not acknowledged

(24)

Alarm History

24

(25)

(26)

Fault Management Features

- The performance values of specific KPIs in the Flexi Multiradio BTS are supervised.

• The set of supervised KPIs is predefined and fixed in Flexi Multiradio BTS. • It does include few (around five) important BTS related KPIs

• The KPI values calculated in Flexi Multiradio BTS will be compared to defined threshold values.

• Overall constraints will be taken into account as e.g. sufficient traffic ongoing to be able to evaluate the situation.

PM – FM Interworking in BTS LTE166, [RL20] 2011

26

• Overall constraints will be taken into account as e.g. sufficient traffic ongoing to be able to evaluate the situation.

• Default threshold values are pre-defined in the Flexi Multiradio BTS, but can be changed by operator They are changeable as parameters in the configuration database.

- If the KPI value does exceed a defined threshold value, an alarm will be raised. - If the value drops under the threshold value the raised alarm will be cleared

again

- If the value drops under the threshold value the raised alarm will be cleared again.

(27)

CONFIGURATION MANAGEMENT

PARAMETERS

PLANS

BACK UPS

UPLOADS

AUDITS

(28)

Configurator: Application for CM

Configurator

Planning Center / 3rd _{party system}

The major components of Configurator

• CM Editor

– Display actual parameter values

– Create, modify and delete plans

• CM Operations Manager

– Upload, download from / to the network CM

CM

Operations CM

Northbound interfaces

28

Radio/Core

– Upload, download from / to the network

– Import, export via northbound interface

• CM Analyzer

– Run consistency checks

– Define user-specific consistency checks

• Southbound interfaces • Northbound interfaces CM Analyzer Operations Manager CM Editor Southbound interfaces Core Rel4 WCDMA Siemens GSM & WCDMA Multi-vendor integration WiMAX & I-HSPA & LTE CM* Nokia GSM

(29)

Basic Operations

Configurator CM

Regional NetAct

Element Manager

Customer’s planning system

Export Import launched Network On the field Configurator CM Upload Provision • download • activate CM events Manager Local operations launched

(30)

CM Essentials - CM Editor

• Editing & browsing through actual and planned configuration data

• Integrated plan management

– For all network elements (BSC, RNC, MSS, MGW etc.)

– Parameter mass modification

– Advanced search

Description

OSS5.1

30

– Advanced search

– Template management

• Operator can compare between actual and planned parameters while editing

• Find objects that match specified criteria with possibility to set new values only to these objects

• Templates to complement planning tool data for minimizing user errors

(31)

Data Management - CM Operations Manager

• Keep Configurator up-to-date with network

– Upload actual configuration

• Efficient plan-based management

– Plan provisioning to network

– Import plans from Nokia Siemens Networks or 3rd party planning

Description

Networks or 3rd party planning environment

– Export actual data and plans

– Compare actual with planned data

• Scheduling of operations

• O&M Operations Manager – all operations under control

• Comprehensive feedback to all

(32)

CM Analyzer

• Consistency checking for

–Planned and actual configurations

–Parameter values

–Dependencies within a network element

–Dependencies between network

Description

OSS5.1

32

–Dependencies between network elements

• Scheduled start of consistency checks

• Automatic corrections of inconsistencies

• Automated reporting that can be sent to e-mail address

• Detection of planning errors including human errors at an early stage

• Higher quality of network plans

(33)

(34)

Major functional changes between NetAct 5 and NetAct 8

Monitoring as the alternative for license key management

NetAct 8:

• Nokia Siemens Networks’ software asset monitoring via SWAM client as an alternative for license key

NetAct 8 _NSN Operator SWAM Client CM, PM database NetAct License Manager 34

alternative for license key management.

• Supported use models: • Pay as you grow • Pay as you configure • Pay as you use

(35)

SWAM Technical description NetAct SW feature activation parameters NSN Operator Monthly reports SWAM Client NetAct CM Database

• Operator configures RAN parameters as per

Radio Network

• Operator configures RAN parameters as per internal SW license investment plan

• NetAct configures the Radio Network as per the input parameters

• SWAM client runs the report query at pre-defined intervals and creates reports of configured/used parameters

• Reports are sent to NSN and operator at pre-defined interval

(36)

LTE Basic SW – Pre-Paid model

• Operator configures the number of Connected UE for each eNB (value equal to the purchased license for BSW)

• The maximum number of Connected UEs per eNB for Flexi Multiradio BTS is controlled by a configuration parameter “maxNumActUE”

• Connected UEs exceeding the configured value of the parameter are rejected

• SWAM client gathers the value of this parameter for each eNB

periodically and CT uses SWAM report to ensure the adherence to the

36

periodically and CT uses SWAM report to ensure the adherence to the contractual agreement 20 15 10 50 30 75 NetAct Configured value of Connected UEs Dense Urban suburban Rural eNB 75

Highest peak value allowed

At any given time 76th _{user will}

be rejected by scheduler

CU

(37)

LTE Basic SW – Post-Paid model

• The maximum number of Connected UEs per eNB for Flexi Multiradio BTS is stored by PM counter (ENB_LOAD_ACT_UE_MAX) in NetAct • SWAM client gathers the value of this PM counter for each eNB

periodically and compares it to previously stored highest value

• If there is a positive change in the value, the delta value is reported and CT will invoice the customer for this delta value per eNB

• The past highest value is replaced by the current highest value and stored in the database for the future report comparison

stored in the database for the future report comparison

20 15 10 50 30 75 NetAct Counter value of Connected UEs during measurement period Dense Urban suburban Rural _eNB

Value 75 is stored as new Highest value to be compared

Counter Period current -1

Period current Invoicable value

Max.Connected Users

(38)

PERFORMANCE

MANAGEMENT

38

(39)

3. LTE RL40 measurements

RNW Measurements

LTE S1AP (M8000) LTE Cell Load (M8001) LTE Transport Load (M8004) LTE Power and Quality UL (M8005)

LTE EPS Bearer (M8006) LTE Radio Bearer (M8007)

LTE RRC (M8008)

LTE RL40 measurements are divided into two subgroups as presented bellow:

Transmission Measurements RNW Measurements

LTE Inter System Handover to GSM per Neighbour Cell (M8019) LTE Cell Availability (M8020)

LTE Handover (M8021) LTE X2AP (M8022)

LTE UE and Service Differentiation (M8023) LTE RRC (M8008)

LTE Intra eNB Handover (M8009) LTE Power and Quality DL (M8010)

LTE Cell Resource (M8011) LTE Cell Throughput (M8012)

LTE UE State (M8013) LTE Inter eNB Handover (M8014) LTE Neighbour cell related Handover (M8015)

LTE Inter System Handover (M8016)

LTE Inter System Handover to UTRAN per Neighbour Cell (M8017) LTE eNB Load (M8018)

Transmission Measurements LTE IP Statistics (M51120)

LTE PHB Statistics (M51121) LTE Ethernet Link (M51123)

LTE IP Sec (M51125) LTE IP Filtering (M51126) LTE VLAN IP Statistics (M51127) LTE VLAN PHB Statistics (M51128)

LTE VLAN Statistics (M51129) LTE ETH_IF Statistics (M51130)

(40)

4. Pre-defined reports overview ->

System Program

• System Program gives a general overview of network status. With this report the user can achieve in high level how network is performing and understand if problems are occurring. This report should be used to benchmark the network.

These reports include NSN LTE RL30 benchmark KPI's, such as: • Cell Availability

• RRC Connections

• Radio Bearer

• E-RAB

• Data Volumes and Throughputs

• HOs and mobility analysis

• RLC Retransmissions and HARQ

•

40

•Active Ues

(41)

(42)

4.

Pre-defined reports overview ->

Accessibility and Retainability

• Accessibility and Retainability subset provides detailed analysis and root failure causes for problems when trying to access the network or troubleshooting of drop calls/services. These reports should be used to understand why calls are being rejected or dropped.

• The subset is divided into four reports: • Accessibility and Retainability

• Network Access - General and S1AP

• Network Access - Service Access and RRC

These reports include metrics for the following issues:

• Attach and Setup SR

• Initial Context Setup

• S1 interface

• Paging

• E-RAB access fails per cause

• RACH stp SR

42

• Network Access - Service Access and RRC

• Service Retainability • RACH stp SR

• RRC Access fails per cause

• ECM idle to ECM Conn transitions

• RRC releases per cause

• Data RB releases per cause

• E-RAB normal releases

• E-RAB drops per cause

(43)

4. Pre-defined reports overview -> Mobility

• Mobility subset provides handover and mobility statistics. Decision rate, Intra eNB, Inter eNB and Neighbour HO performance are available. These reports should be used in the optimization of neighbors lists. It should also be used to analyze the performance of HO analysis for UTRAN and Network Assisted Cell Change for GSM per neighbor.

These reports include metrics for the following issues for intra and inter HO:

• HO Decision Rate

• HO SR

• HO attempts

• HO preparation SR

• HO Prep FR per cause

• HO Drop Ratio per cause

• HO Preparation, attempts and fails per Neighbour cell

• CS Fallback

• NACC

(44)

4. Pre-defined reports overview ->

Interface analysis

• Interface Analysis subset provides statistics of X2, IP and Ethernet Interfaces. Metrics per VLAN and per QoS Class are also available in these reports. These reports should be used to analyze if throughputs and data volumes are the expected ones in X2, IP and Ethernet.

This Subset is divided in six reports:

•X2-Interface

Control Plane data (incoming/outgoing throughput and data volume)

User Plane data (incoming/outgoing throughput and data volume)

•IP interface traffic

Incoming/Outgoing volume

Incoming/Outgoing dropped congestion

44

Incoming/Outgoing dropped congestion

•IP interface traffic and QoS classes PHB

Incoming/Outgoing data volume and throughput per QoS Class

•Ethernet Interface

Inbound/Outbound data volume/throughput Received Ethernet frames SR

•VLAN analysis per QoS class

Inbound/Outbound data volume and throughput per VLAN and per QoS class

•VLAN total volume analysis

(45)

4. Pre-defined reports overview -> Modulation and

Coding Scheme

• Modulation and coding scheme subset provides statistics of data volumes, transmissions, share of transmissions and successful transmissions per MCS. This reports should be used to understand the quality of data volumes transmissions and DL Adaptive

Modulation and Coding (AMC) algorithm performance to improve system capacity and peak data rate.

• Transmissions and Volume per Low/High MCS

PDSCH and PUSCH transmissions using low/high MCS

Percentage of DL/UL traffic volume using Low/High MCS codes

• Percentage of transmissions per MCS

PDSCH and PUSCH PDSCH and PUSCH

• PUSCH 1ST _{transmission per MCS}

• Unsuccessful Transmissions per MCS

• NACK MCS time division

•Data Volume per MCS

MAC PDU volume on PDSCH/PUSCH per MSC

(46)

4. Pre-defined reports overview -> Integrity

• With Integrity subset is possible to analyze delays, congestions and encryption/decryption metrics. These reports should be used to understand delays in

different protocols levels.

• IP Packet integrity

ESP packets encryption SR

Dropped packets due to ingress rate limiting Discarded packets due to Filtering violations

• Packet delay

PDCP SDU delay UL/DL (min, max and avg) DL/UL SCH TB Retransmissions and error ratio RLC PDU Retransmissions

46

RLC PDU Retransmissions

• BLER per Modulation

QPSK, 16 QAM Gain using HARQ

•Congestion per QoS classes for IP PHB

IP PHB Packets to send drop rate and data volume lost due to congestion per QoS class

•Congestion per QoS classes per VLAN

IP VLAN PHB Packets to send drop rate and data volume lost due to congestion per QoS class

• IPsec packets

(47)

4. Pre-defined reports overview -> Usage

• With Usage subset is possible to analyze throughputs, resources utilization, power and number of active users.

• With these reports, throughputs at different levels should be checked to see if it matches the network configuration. They are also very important to understand the Physical Resource Blocks utilization for issues like network upgrades, power analysis with downgrades/upgrades in scheduler and MIMO

This Subset is divided in seven reports: •Throughput at PDCP level and per QoS class

PDCP UL/DL thp (min, max and avg) PDCP SDU THP and data volumes QCI

• Physical Resource Block utilization PRB utilization UL/DL

DL/UL PRB Analysis per TTI, per UE DL/UL PRB utilization level per TTI

• RLC, SDU, RRC, analysis

RLC SDU data volume per channel MAC SDU volume per channel

with downgrades/upgrades in scheduler and MIMO performance. They are also very important to understand the throughputs per Quality of Service Indicator to check the distribution of network resources per group of users priority.

MAC SDU volume per channel RRC UL/DL Traffic volume

• Power

UE Power HeadRoom for PUSCH per level

UE Power analysis (scheduler upgrades/downgrades)

UE Delta Tx PUSCH/PUCCH per level

• MIMO

Open Loop Diversity/Spatial Multiplexing Mode Usage and Switches

(48)

TROUBLESHOOTING

48

(49)

PROBLEMS ENCOUNTERED IN LIVE NETWORK LAST 1.5 YEARS RL20+

• RP3 ALARMS

• AUTO CONFIGURATION

• ANR CREATE TOO MANY ADJ

• PCI MANUAL PLANNING MISTAKES

• NETWORK OUTAGE CAUSED BY MME PARAMETER CHANGE

• NETWORK OUTAGE CAUSED BY SIG LOAD ON S1 DUE eNB MASS SW Ug • NETWORK OUTAGE CAUSED BY SIG LOAD ON S1 DUE eNB MASS SW Ug • SLEEPING CELLS (Related to RP3 Delay)

• CONFIGURATION CHANGE AFTER SW Upgrade • UE SW Bugs

• THROUGHPUT ISSUES ,FRAGMENTATION FTIB • LOST VOICE CAPABILITY (CSFB Issues)

(50)

Troubleshooting guidelines – Problem type according to KPI classification

Problem type according to main KPI categories:

- Accessibility problems:

• Service accessibility, e.g. Attach, or EPS Bearer Setup Request success rates

- Reliability problems:

• Once established, the reliability of the service, e.g Voip call, or EPS bearer drop rates

50

drop rates

- Integrity/Error rates problems:

• Promised to deliver the service with a certain quality. E.g. Cell throughput, or user data rates, packet delay.

- Availability problems:

• Relationship between service up and down times, e.g. cell, or LTE Network availability

- Mobility problems:

• Different mobility related metrics, e.g. hand-over success or drop rates, latencies.

(51)

Troubleshooting guidelines

- The above-mentioned cases it is essential to try to get at least following logs : • eNB snapshot

• Interface traces S1-C by wireshark

- The eNB is connected to the EPC (S1 interface) by IP switches. The

precondition for capturing traces on these interfaces is the availability of a mirror port on one of the switches. The tracing on these interfaces can done either using wireshark (usually for lab tests) or using commercial protocol

51

either using wireshark (usually for lab tests) or using commercial protocol trace equipment (e.g. NetHawk M5, JDSU SART). If the IP traffic on these interfaces is encrypted (e.g. if IPSec is used) it is necessary to use the deciphering functionality of the protocol tracers.

• Emil logs

- BTS message remote tracing: BTS message remote tracing can be configured, started and stopped. Depending is tracing covering only Control Plane or also User Plane messages, estimated 70-500 3 sector LTE BTS can be traced at the same time with one PC.

• UE trace ( e.g. by using Nemo tool) • UDP log file by BTSLog tool NWS LTE RA RD eNB I&V

(52)

Fault sources

• SW

- eNB SW bugs

• SW not working according to specifications

• Specifications are ambiguous or lead to faulty behaviour - SCF parameters

• Suboptimal parameters, missing neighbour relations, etc. - UE SW bugs

• UE is behaving against specifications or against NSN expected behaviour - EPC SW bugs

52

- EPC SW bugs

• MME/S-GW is behaving against specifications or against NSN expected behaviour - Backhaul SW bugs

• Backhaul is dropping packets or sending with too high latency/bursts

• HW

- Faulty eNB units - Faulty/instable UEs

• Commissioning

- Wrong antenna tilting, loose cables, wrong commissioning parameters, wrong/missing synch sources

• Network planning

- Too large cells, gaps in geographical coverage, high interference sources, blocking buildings, high mobility Tracing Areas, etc.

• Diagnostics

(53)

eNB SW bugs

• Symptoms

- No eNB L3 reply for visible UE/EPC signalling - No eNB L3 messages sent when expected

- Rlc Ack visible in call drop after RRC Connection Release - RLF or Errors visible in good signal conditions

- Call drops without any RLF/Errors - Huge amount of Ghost Calls

- Sudden drop of multiple calls at the same time - Sudden drop of multiple calls at the same time - Sudden drop in KPI without parameter changes - Sudden drop of KPI after new feature activation - Sleeping cells

- HO rejections

- RRC connections are reject

- KPI drop correlation with snapshot fetching, SiteManager connectivity or other normal maintenance actions

- DSP crashes

• First step troubleshooting actions - Analyze Emil/UE Traces

- Set triggers and analyze from snapshot TTI Trace and SysLog for unusual behaviour and signal conditions

(54)

SCF parameters

• Symptoms

- No UE reply for RRC signalling for multiple UE types

- RACH preambles are missed in HOs

- Poor signal quality and RLF failures or Errors indicated in certain call

phases

- Errors distributed evenly in the network

- PDCCH blocking visible in counters

54

- Errors distributed evenly in the network

- PDCCH blocking visible in counters

- Timers expire

- High interference visible

- Sudden drop of KPI after parameter changes

- Call drops correlate to active UE amount

- HARQ/RLC/Preamble Retransmissions reach maximum value

- Measurement reports don’t initiate HO preparation

- HO are started in too poor source cell signal conditions

• First step troubleshooting actions

- Analyze KPI, PM counter and SCF parameter content and

(55)

Call drops UE SW bugs

• Symptoms

- No UE reply for RRC signalling for one specific UE type

- A lot of RLF or Error visible for one specific UE type

- KPI drops after new UE type is launched in the network

- KPI drops after new UE SW version is launched in the network

- UE does not reply in good signal conditions and low loaded cells

- No recovery method for dongle disconnect (No Detach messages

visible)

- Collect UE logs if possible

(56)

EPC SW bugs

• Symptoms

- No reply to S1AP messages

- Sudden S1AP Failure indications to multiple eNB

- S1 Error indications/Alarms to multiple eNB

- Missing S1 HO command forwarding to target eNB

- Delayed reply/forwarding to S1AP messages

- UEContextReleaseCommand without strange cause values

56

- Delayed reply/forwarding to S1AP messages

- UEContextReleaseCommand without strange cause values

- Duplicate/Strange MME UE S1AP ID allocations

- KPI drops after new EPC SW upgrade

- KPI drops after EPC parameter upgrade

- FTM mirror port Wireshark trace

- FTM logs

- EPC logs if possible

- In case NSN EPC, contact NSN R&D

(57)

Backhaul SW bugs

• Symptoms

- KPI drops after new Backhaul element type/ SW version/feature is

launched or activated in the network

- KPI drops after Backhaul parameters/routing version is launched in the

network

- Link breaks for a cluster of eNB (e.g. behind a specific router)

- S1 Error indications/Alarms for a cluster of eNB

- Link breaks for a cluster of eNB (e.g. behind a specific router)

- S1 Error indications/Alarms for a cluster of eNB

- Increased latencies in TCP level

- Increased TCP re-transmissions

- Congestion alarms/indications

- Connectivity problems to certain IP addresses/TCP ports

- Throughput can not exceed certain (low) limit

- FTM mirror port Wireshark trace

- FTM logs

(58)

Faulty eNB units

• Symptoms

- Raw or Customer Alarms indicating faulty/missing units

- Known problematic HW versions are installed in the field

- KPI drops after new HW installations

- KPI drops after storms, floods, earth quakes, etc. natural disasters

58

- KPI drops after storms, floods, earth quakes, etc. natural disasters

- Call drops located under certain cell away from cell edge

- Long periods/Frequent cell unavailability

- Frequent site resets

- Check eNB status with SiteManager

- Arrange HW swap

- Check that there is no systematic SW/Comissioning/HW version fault

(59)

Faulty UE

• Symptoms

- On UE type or user in same location is causing frequent call drops

- Call drops follow similar pattern in same cell location and during time

of day

- High interference peaks in same cell location and during time of day

- Bursts of call drops in one cell for short time period, without the cell

being sleeping/disabled

- Try to find out problematic UE type

- Try to find out problematic subscription IMSI

(60)

Commissioning

• Symptoms

- HO fail in target cell for specific eNBs -> frequency sync faulty or missing

- Line code violation (and high amount of ghost calls) -> problems in

RP3-01 optical fibres

- Antenna line alarms - > Loose feeder cables

60

- Antenna line alarms - > Loose feeder cables

- No S1 connection -> Wrong IP settings

- Start-up problems or alarms-> Invalid commissioning parameters or

unsupported configuration

- Call drops inside certain cell area out side cell edge -> antenna tilting

- High DL interference -> Wrong cell power settings or antenna tilting

- Check status with Site Manager

(61)

Call drops

Network planning

• Symptoms

- There are clear holes in network coverage, when analyzed with NetAct Optimizer (or other application, which displays cell locations)

- There are clear non-line of sight locations for dense populated places

- Cell under Train, boat or bus traffic are configured with basic HO parameters - eNB indicates lack of resources

- eNB indicates lack of resources

- Scheduled UEs per TTI is near maximum continuously during busy hours - Timing Advance values indicate too far distances for used carrier frequency

and fading environment (e.g. urban city with tall buildings) - Call drops in cell edge and no meas report sent

- Signal levels are low in large cell areas

- Frequent call drops in same location with multiple UE types - High DL interference is certain cell locations

- Check status with NetAct Optimizer or site plan - Contact customer for poor coverage

(62)

Call drops Diagnostics

• Symptoms

- Contention resolution timer expires and some kind of tracing is active

- CPU loads are high (> 80 %) despite of moderate Traffic mix

- Heavy amount of prints in SysLog or short Snapshot SysLog duration

- Emil trace activated during heavy CP traffic

62

- Emil trace activated during heavy CP traffic

- Strange R&D flags visible in swconfig.txt or in Start-up syslog prints

- UP data traced/uploaded loaded continuously

- Backhaul Traffic >> Air Interface traffic

- Turn of tracing and additional logging

- Switch of R&D flags

(63)

(64)

Data Collected from eNodeB for Analysis

- The eNB snapshot contains the following information: • Active alarms and alarm history for this eNB

• Installed SW versions

• eNB configuration (commissioning parameters)

• Internal log files for different eNB components (including MAC TTI traces (10 MB))

BTS Technical Report/BTS Snapshot

64

MB))

• PM counter data for this eNB (starting from the last eNB reset, but max for 24 hours)

- Logs are frozen by eNB when fetch command is received - Saved in .zip file when loaded

- Collection of technical log is triggered by via the

• BTS Site Manager protocol. When technical log is collected from the Site Manager, the term snapshot is used as a synonym.

(65)

Measurement Tools (NSN R&D Internal tools)

The following measurement tools have been developed by NSN for R&D that you can also use • BTSLog

• BTSlog is a testing tool including functionality of:

• System log recording & real-time monitoring tool • Message trace and recording

• R&D parameters handling

• BTSlog is used for BTS testing and works with Windows • LTE Browser

• LTE Browser is used to get the real time data about the CPU load, & real time UL PHY measurement

• LTE Browser is used to get the real time data about the CPU load, & real time UL PHY measurement data (SNR, BLER etc) memory dump information's from DSPs and static Cell specific configuration information

• Emil

• EMIL is an offline message tracing analyzer tool.

• EMIL parses all the internal messages between different software components,

• EMIL has the capability to show all ASN.1 messages (RRC, S1 AP, X2 AP messages) • Emil L3 decoding works even if IPsec is used for S1/X2

• TTI Trace Analyzer

• Traces Cell and User specific TTI information

(66)

66

(67)

Analysing BTS message logs with Emil

(68)

LTE BROWSER

• Definition: noise rise NR = (I+N)/N, I = interference, N = thermal noise

• SNR0=S/N (only noise, no interference)

• SNR1 = S/(I+N), (noise + interference using eg external rf generator)

• NR = SNR0/SNR1 = (I+N)/N, assuming average signal power S is the same • Example: SNR0=13dB, SNR1=8dB NR= 5dB

SNR0

Real-time online measurement from eNB

LTE Browser

68

SNR0 SNR1 Interference source is on Noise rise Interference source is off (only thermal noise)

(69)

TTI Trace Analyzer

(70)

UL HARQ Support in TTI Trace

8 HARQ processes per UE

[3GPP-36.312 chapter 8] specifies the relationship of the timing between eNB and UE with the following wording: “For FDD and normal HARQ operation, the UE shall upon detection of a PDCCH with DCI format 0 and/or a PHICH transmission in subframe n intended for the UE, adjust the corresponding PUSCH transmission in subframe n+4 according to the PDCCH and PHICH information”.

Predefined RV sequence for successive packet transmissions

ToC

70

Retransmission is sent for Nack or DTX exactly after 8TTIs. This is synchronous HARQ procedure. Unlike DL , HARQ process id not sent in UL because the UE is

expected to retransmit in exactly 8 TTI. ACK/NACK is sent by eNB in exactly 4 TTIs.

Non-Adaptive HARQ retransmissions use the same MCS, the same rate matching and the same PRBs

RRM in TTI traces & related System Performance - Shomik Pathak - Nov 2011

(71)

(72)

Traffica for eNodeB, what and how?

• Traffica for eNodeB provides

• Real-time monitoring solution for subscribers radio interface experience • Direct interface to NSN Network element

• Subscribers radio interface experience

• Includes detailed internal and external error causes for root cause analysis

How Traffica for eNodeB works

72

How Traffica for eNodeB works

• Traffica receives data reports based on actions Users Equipment in the network.

• User Equipment specific report is sent in real time immediately to Traffica when the triggering event happens.

• RRC/eRAB activities by User Equipment will trigger a data report to sent to Traffica.

• RRC or eRAB establishment/release and related failures.

• Handover and Handover attempts will trigger a data report to be sent to Traffica.

(73)

Traffica for eNodeB, use cases

• Real-time view to Radio events

– Detect Radio failures as they occur

– Visibility to events, locations, failure

causes

– Locations with most events or failures

– Monitor Number of call setups / cell

• Visibility Handovers

– Monitor Handover events

• Drive Test Minimizing

– Cell-ID and UE distance from eNodeB

included in the Traffica report, UE location can be specified quite well.

– At handover, UE measurements from

each neighbour cell (field strength) is available

– Traffica can count normal and

abnormal call setups and releases

– Monitor Handover events

– Visibility to Handover failures and

causes

– Handover locations and field strength

• Location based analysis

– events and failures per location

abnormal call setups and releases according to each location

Fast root cause analysis via detailed internal error cause descriptions

(74)

Data Collected from eNodeB for Analysis

- Traffica receives event based reports from Trace Data Collector in real time.

- Traffica analyses the data and creates KPIs based on the data and visualize in Traffica real-time graphs.

General (graph set):

• RRC overview (setup failure, active failure, setup attempts, normal releases, setup failure distribution, etc)

• RRC service failure Details(RRC setup events, RRC setup events / establishment cause, RRC setup failure distribution /cause) • RAB overview (setup failure, active failure, setup attempts, normal releases, setup failure distribution, etc)

• RAB service failure Details(RAB setup events, RAB setup events / establishment cause, RAB setup failure distribution /cause)

• Handover Analysis (graph set)

LTE1053 Traffica for LTE

74

• Handover Analysis (graph set)

• Incoming HO types (Incoming HO per cell, Incoming HO types, UE distance measurements, source cell RSRP, Target cell RSRP)

• Outgoing HO types (outgoing HO per cell, outgoing HO types, UE distance measurements, source cell RSRP, Target cell RSRP)

• Handover failures (outgoing HO failures / cell, HO failure causes, HO failure phases/cell)

• Troubleshooting (graph set)

• RRC setup failures(RRC setup failures per cell, RRC setup failure sources /cell)

• RAB setup failures (RRC setup failures per cell, RRC setup failure sources /cell, UE distance at RAB setup failure) • RAB active failure(RAB active failure per cell, RAB active failure sources per cell, UE distance at active failure) • Handover failures (outgoing HO failures / cell, HO failure causes, HO failure phases/cell)

• Triggers for receiving these reports are

• RRC or RAB establishment / release and failures related to those

• Intra-system or Intersystem handover, successful or failure includes failure cause information. • The interface between the Trace Data Collector and Traffica is event reports based (TCP/IP).

(75)

NetAct Tracing for LTE Introduction

- NetAct Trace Viewer provides the tracing functionality for LTE

- The same application can be used for LTE tracing as for 2G and 3G tracing.

- LTE tracing allows to monitor calls of subscribers in LTE network - Functionality aligned with Call Trace Analyzer

NetAct

Trace Viewer

eNB

(76)

NetAct Tracing for LTE Use Cases

- Network Rollout and new installation

• Monitor calls of field engineers on-line

• Guide drive testing team at the problem spot on a field instead of testing just

pre- planed route

• Get fast reply after software and hardware upgrade about network coverage

by monitoring on-line field engineers

• Act promptly to corporate customer network coverage related problems

• Find out right problem related to case

76

• Find out right problem related to case

• Reduce time used to network coverage related cases

- Detailed investigation of subscriber complaints

- Solving problem cases related to network coverage - Features and trace content for

• Site & Cluster acceptance • Change acceptance

• Saving test results • In depth analysis

(77)

Measurement Tools – Terminals (Drive Testing)

• The following drive testing tools using a terminal are recommended:

• Agilent/JDSU E6474A (Currently most cost efficient solution and supports samsung)

• Currently support LG Data card and partial Samsung support • Support soon also Qualcom Chip set

• Supports up to eight frequency bands

• NEMO Outdoor from Anite (No samsung support)

• LG LTE L-1000 and L-2000 chipset support • LG G7, G13 and G17 UE support

• PCTEL EX LTE Scanner and R&S TSMW LTE scanner supports

• Support soon also Nokia Modem RD-3 and QUALCOMM LTE chipsets (MDM9200 / MDM 9600)

• Rhode and Schwarz (ROMES)

R&S®ROMES4 is a test platform for mobile measurements in all modern radio networks. In combination with scanners and test mobile phones, it forms a complete system for coverage and quality-of-service (QoS) measurements.

• Accuver XCAL

(78)

https://twiki.inside.nokiasiemensnetworks.com/bin/view/LTETechSup/WebHome LINK FOR LTE TOOLS

LINK FOR LTE TOOLS

78

LINK FOR LTE TOOLS LINK FOR LTE TOOLS

(79)

SON FOR LTE

• Flexibility in logistics (eNB not site specific)

• Increased network quality and perform.

• Error Self-detection and mitigation

Self-Healing Self-Optimization

Self-Configuration

(eNB not site specific) • Reduced site /

parameter planning • Simplified installation

- less prone to errors • No / min. drive-tests • Faster roll-out and perform. • Parameter optimization • Reduced maintenance, site visits mitigation • Speed up maintenance • Reduce outage time

(80)

• SON Plug and Play

• Flexi BTS auto-configuration

• Automated Physical Cell ID assignment

• Automated Neighbor Relations (ANR) Management

• Automated neighbor list configurations of LTE, inter-frequency LTE, inter-system WCDMA and GSM neighbor cells

• Automated update of neighbor list configuration in case of GSM Centralized

SON

Self-Configuration for LTE

80

• Automated update of neighbor list configuration in case of GSM and WCDMA neighbor cells removals / additions

• PRACH Management

• Automatic assignment of PRACH parameters by Optimizer Distributed

SON

• SON Plug and Play

• Flexi BTS auto-connection

• Flexi BTS auto-configuration

• LTE Certification Authority and Registration Authority Management

• Automated Neighbor Relations (ANR) – Fully UE based

• LTE, inter-system neighbor list configuration due to UE found/measured neighbor relations

(81)

• Automated Neighbor Relations Management

• Optimization of found neighbor relations by blacklisting inefficient ones

• Energy Saving

• Centralized management for Flexi BTS shutdown and energy saving features

• Mobility Robustness Optimization

• Optimizing Intra-frequency radio network HO-configuration in order to reduce too early, too late, ping pong HOs or HO to wrong cells

• Service Quality Optimization

Self-Optimization for LTE

Centralized SON

Distributed

81

• Service Quality Optimization

• Minimization of Drive Tests

• Geographical analysis of location specific performance based on call level statistics

Distributed

SON • Automated Neighbor Management• Support for optimization of neighbor relations

• Energy Saving

• Intelligent Flexi BTS shutdown

• Enhanced power saving

• BTS power saving mode

• Load Balancing

• Physical HO optimization

• Real-time load balancing

(82)

• Cell Outage Detection

• Cell outage detection by KPI and alarm supervision

• Cell Outage Triggered Reset

• Cell outage detection triggered reset of cells

• Cell Outage Compensation

• Coverage hole compensation by antenna adjustments Centralized

SON

Self-Healing for LTE

82

• Coverage hole compensation by antenna adjustments

• Automatic Interface Alarm Correlation

• Alarm correlation in case of problems of a central interface (X2, S1) or entity

Distributed SON

• Cell Outage Detection and Triggered Reset

• KPI and alarm support

• Cell Outage Reset

• Automated cell lock & unlock with escalation to reset

• Cell Outage Compensation

(83)

Optimizer in the core of the Self Organizing Network solution in NetAct for centralized SON

Self-Optimisation:

▪ Auto-tuning of neighbour cell lists and other key parameters based on UE and eNB measurements on network management level

Optimizer for LTE

Domain Network Management Optimizer NetAct PM CM Optimizer CM

Effortless self-configuration and optimization on Network Layer to maximize the network performance

BACKRL40

on network management level

▪ Cell outage compensation

▪ Able to cope with a single vendor domain as well as with multi-vendor access networks

Domain Manager PM CM Nokia Siemens Networks Vendor X

(84)

Physical-layer Cell ID (PCI) Management

- PCI management done in similar way than UTRAN PSC - PCIs are not unique

- initial allocation of PCI values

• confusion-free: all neighboring cells need to have different PCI values • collision-free: no two cells must be received with same PCI

- minimum reuse distance algorithm PCI group, N(1) ID PCI offset, N(2) ID 0 1 2 0 0 1 2 84

- minimum reuse distance algorithm

- optional: neighbor’s neighbor have different PCIs • Support for synchronized LTE networks

- three additional rules to allocate PCIs

• cells under one eNB shall have PCIs from the same PCI group(s) • neighbor cells shall avoid same PCI modulo 30

• neighbor cells shall use odd and even PCI groups evenly (approximation of a modulo 6 rule) Achim 0 0 1 2 1 3 4 5 2 6 7 8 … … … … 165 495 496 497 166 498 499 500 167 501 502 503

(85)

PCI vs. eUTRAN Cell Global ID (ECGI) eUTRAN Cell Global ID

• Part of System Information Block 1 in BCCH

• Unique Cell Identity in the with PLMN and eNB Identities • Long repetition cycle of 80 ms

• UE does not decode this if not advised to do !!!

Physical-layer Cell ID

• Part of the Synchronization Sequence/Reference Symbol • Limited to 504 values, not unique

• Short repetition cycle of 1 ms • fast, no DRX needed

• careful assignment needed because a UE shall never receive the same value from 2 different cells -> no collisions, no confusions

(86)

Physical-layer Cell ID (PCI) Management

- Need to have clear identity codes to ensure reliable neighbour cell dependencies

- Similar structure/management to UTRAN Primary Scrambling Codes

- PCI allocation algorithms:

• confusion/collision free allocation

23 23 14 14 1 1 66 66 412 412 511 511 234 234 322 322 98 98 86

Author / Date

• confusion/collision free allocation

• minimum reuse distance in allocation criteria • user can control the algorithm

- option to allow neighbor’s neighbor have different PCIs

- minimum reuse distance can be defined

• support for synchronized LTE networks

- cells under a eNB shall have PCIs from the same PCI group(s) - 504 PCIs have been grouped into 168 groups, 3 codes in each - neighbor cells shall use odd and even PCI groups evenly

- Optimizer 3.0 assigns initial PCI values in auto-configuration process

(87)

LTE PCI Management Enhancements

- If LTE eNodeBs are deployed and managed with various processes it can happen that PCI collisions or confusions occur

• When LTE Auto-configuration process is used for eNodeB roll-out PCI conflicts never happen

• 23 23 14 14 66 66 412 412 511 511 234 234

• PCI conflict/confusion analysis can be activated in Optimizer

– Collision analysis for selected set of cells

–• Collisions and confusions checked against the user defined criteria

1 1 66 66 234234 322 322 98 98

– Collisions and confusions checked against the user defined criteria

– Analysis against violations of additional constraints is run too

Forbidden PCIs (in Optimizer 3.0 CD2)

– Analysis results are presented in table views and on geographical map

• PCI reallocation can be activated in Optimizer

– Resolution can be activated for the selected area

– Optimizer finds the collision/confusion and violation

free allocation for PCIs by minimising the amount of changes

(88)

Automated Neighbor Relations (ANR) in LTE

• The handset mobility relies on information given by neighbor cells

and the quality of it

• The ANR function automates and optimizes that handover

process with real-time information exchange between the parent

cell (source) and receiving cell (target)

• The automated neighbor relations remove manual planning,

88

The steps of ANR process

• Neighbor cell discovery

• Neighbor Site’s X2 transport configuration discovery

• X2 Connection Set-up with neighbor cell configuration update

• The automated neighbor relations remove manual planning,

maintenance and subsequently errors from the process

(89)

• The automated detection and configuration of unknown

cells and respective sites supports the self configuration of

the neighbor cell information without operator involvement

Automation of adjacency planning & maintenance

leads to O&M cost reductions The error free ANR process

simplifies and speeds up the network rollouts and

Key Benefits of ANR

simplifies and speeds up the network rollouts and

expansions for faster time to market

Expected amount of LTE neighbor sites is 6-32

(90)

ANR support for LTE with OAM extension (LTE492)

ANR support for LTE with OAM extension (LTE492)

new mechanism allowing for identification of unknown UE-reported neighbour cells

each newly activated Flexi Multiradio BTS is provided with table that maps Physical Cell Id & DL EARFCN of adjacent cells to C-plane IP address of their parent eNBs

90

address of their parent eNBs

PCI/RF/IP@ mapping tables are generated by NetAct based on the neighbour site relations analysis and downloaded to BTS during auto-configuration procedure

based on PCI/RF/IP@ mapping table, eNB is able to determine adjacent eNB hosting unknown cells and to retrieve from it necessary (in the context of neighbour relations) configuration information

neighbour site and neighbour cell configuration information is derived via X2 connection

(91)

ANR support for LTE with OAM extension (LTE492)

ANR support for LTE with OAM extension (LTE492)

X2 connection setup procedure is triggered by UE measurement reports containing PCI that is not known to eNB as a neighbour cell but is listed in PCI/RF/IP@ mapping table

X2 connection establishment is not based on info from MOC ADIPNO only as it was with LTE724

target eNB accepts incoming X2 connection setup requests from a source eNB even if no IP-address requests from a source eNB even if no IP-address of the source eNB is configured on the target site

no need for configuration of IP@s at both ends of the X2 connection as it was with LTE724

Feature benefits

automated detection and derivation of configuration information of unknown neighbour sites and cells

no need for manual pre-planning of eNB adjacencies for X2 connection setup self-configuration of the neighbour cell information without operator

(92)

LTE 771 Optimization of neighbor relations

• NetAct Optimizer supervises all registered cell relations between neighboring LTE cells if they are still valid and reliable candidates to be a handover destination

• When the outcome results in an inefficient

Optimizer Configurator

CM

PM

Evaluating of handover behaviour – blacklisting proposed ?

92

• When the outcome results in an inefficient neighbor relation, the according cell relation is

blacklisted for handover

• Blacklisting can be done manually by the operator as well

• White listing (excluding optimization) can be permanently done manually by the operator • The changes, proposed by optimization

algorithms, can be taken over automatically (without any operator intervention) or after confirmation by the operator

(93)

ANR InterRAT UTRAN / GERAN

Automated configuration: NO manual work for neighbor cell

relation information

RL30

•

Central neighbour relation establishment with NetAct

Optimizer

•

Based on geo-location: no NetAct _Optimizer Configurator CM

•

Based on geo-location: no ANR InterRat supporting UEs

needed LTE GERAN UTRAN UTRAN UTRAN/GERAN Domain Managers N-itf CM GERAN

(94)

Main Menu

Technical Details

• ANR Inter-RAT in details – 1/8

LTE783 ANR Inter-RAT UTRAN / LTE784 ANR Inter-RAT GERAN

LTE site/cell (source of NBR) and RAT site/cell (target of NBR) must be under management of NetAct

For both (source and target), planning information must exist in

NetAct Configurator database, i.e.: geo-location and antenna

direction data

NetAct Optimizer uses this planning data to calculate:

Distance between LTE cell and RAT cell

NetAct OptimizeOptimize rr CM CM LTE Domain LTE Domain Managers Managers CM Configurator Configurator Table of Contents 94

Distance between LTE cell and RAT cell

Is estimated based on geo-locationof LTE cell’s antenna (source) and geo-locationof RAT cell’s antenna (target)

Relative main lobe directions

Is determined based on antennas directionsof source and target cell

Based on them (distance part and relative main lobe directions part),

priority for each adjacency is calculated. Prioritygives information

about usefulness of each NBR.

Also following inputs must be defined:

Search distancebetween LTE cell and RAT candidates for excluding far away cells

Minimum/maximumnumber of NBRs from each considered LTE cell

Otherwise default values are considered

This information must be available asprecondition before starting of NBR planning by LTE783/784 UTRAN UTRAN LTE LTE CM UTRAN/GERAN UTRAN/GERAN Domain Domain Managers Managers source target GERAN GERAN target

(95)

Main Menu

(96)

Mobility Robustness Optimization Input/Output

PM counters

• HO Attempt/Success Rate • RLF rate

Dedicated new counters (based on 3GPP Rel9 Radio Link failure indication and handover report messages on X2-interface)

Input Parameters

96

Output parameters:

and handover report messages on X2-interface)

• Too early HO of type 1 counts attempted failed handovers with subsequent reestablishment

• Too early happening HO of type 2 counts several kinds of problems in handover target cell quickly after a completed handover

• Too late HO counts missed handover attempts ending up in radio link failure

Modified values for

• Cell Individual Offset (CIO) (per NR pair)

(97)

Recommendation to use SON

1. Consider SON expert from the beginning of roll-out procedure plan to

check the advantage of SON.

2. Set-up the correct communication line between global and local.

Involvement of SON expert and product line is strongly recommended to select the SON solution proper and optimized for customer network

environment. environment.

3. Need the engineers who get end-to-end training about SON(not only

eNB SON and NetAct optimizer) because SON issues should be investigated end to end.

(98)

Thank

you

98

Lte Operations and Maintenance