Emil Tool for LTE field debugging
Confidential
Confidential
LTE field Troubleshooting
LTE field Troubleshooting
Current situation
Current situation
•
•
Main sources for t
Main sources for troubleshooting information
roubleshooting information
•
•
KPI counters from NetAct
KPI counters from NetAct
•
•
Alarm history from NetAct
Alarm history from NetAct
•
•
UE terminal logs from drive tests
UE terminal logs from drive tests
•
•
Local SysLog traces & Local TTI traces
Local SysLog traces & Local TTI traces
•
•
BTS Snapshots
BTS Snapshots
•
•
Commercial Interface analyzer
Commercial Interface analyzer
•
•
Process for Troubleshooting
Process for Troubleshooting
•
•
Check from KPI counters problematic behaving cells/sites
Check from KPI counters problematic behaving cells/sites
•
•
Check alarms for outages information
Check alarms for outages information
•
•
Fetch snapshots from problematic sites and forward to
Fetch snapshots from problematic sites and forward to R&D to see if
R&D to see if snapshot
snapshot
files contain any useful information
files contain any useful information
•
•
Check protocol traces for fault details in S1 level
Check protocol traces for fault details in S1 level
•
•
Check are any SysLogs available -> If not try to install BTSLog to site and
Check are any SysLogs available -> If not try to install BTSLog to site and
arrange drive tests for getting UE terminal logs
arrange drive tests for getting UE terminal logs
•
•
If logs don’t contain useful information, discuss
If logs don’t contain useful information, discuss with R&D how to
with R&D how to arrange
arrange
additional prints to logs
LTE field Troubleshooting
LTE field Troubleshooting
Current problems
Current problems
•
•
Log sizes
Log sizes
•
•
CPU inside eNB produce several Mbytes
CPU inside eNB produce several Mbytes
logs in one second
logs in one second
•
•
No centralized debugging system can
No centralized debugging system can
handle all logs for multiple BTS
handle all logs for multiple BTS
•
•
Big logs can not be
Big logs can not be distributed easily
distributed easily
across different parties
across different parties
•
•
IPSec
IPSec
•
•
S1 can not be t
S1 can not be traced unless ciphering key
raced unless ciphering key
is available
is available
•
•
NAS ciphering
NAS ciphering
•
•
NAS protocol in encrypted and can not
NAS protocol in encrypted and can not be
be
decoded without encryption key
decoded without encryption key
•
•
Mobility and call trace without IMSI
Mobility and call trace without IMSI
•
•
Many different call and cell identifiers in
Many different call and cell identifiers in
different SW levels, which are difficult to
different SW levels, which are difficult to
map with each other
map with each other
•
•
IMSI is not available in LTE RAN like in
IMSI is not available in LTE RAN like in
WCDMA
WCDMA
•
•
Different kinds of protocols and
Different kinds of protocols and
interfaces
interfaces
•
•
Many different kinds of encoders needed
Many different kinds of encoders needed
•
•
Vo
Volatile
latile information
information
•
•
Information is available in logs only a
Information is available in logs only a
short period of time
short period of time
•
•
Limited transmission and processor
Limited transmission and processor
capacity
capacity
•
•
Only limited capacity can be allocated for
Only limited capacity can be allocated for
debugging
debugging
•
•
Operator owned networks
Operator owned networks
•
•
Operator has to accept tools and related
Operator has to accept tools and related
costs for debugging and troubleshooting
costs for debugging and troubleshooting
•
•
Flat network architecture
Flat network architecture
•
•
Logs and relevant information are
Logs and relevant information are
distributed across whole
distributed across whole L
LTE network in
TE network in
eNB, no central node where to
eNB, no central node where to debug
debug
•
•
Debugging does not cause eNB crashes
Debugging does not cause eNB crashes
or performance degradation
or performance degradation
•
•
Debugging affects multiple SC
Debugging affects multiple SC
•
•
Network debugging effect to eNB/SC is
Network debugging effect to eNB/SC is
not necessarily tested in lab
not necessarily tested in lab
•
•
UP statistics
UP statistics
•
Confidential
Confidential
Security issues
Security issues
Security Risks
Security Risks
•
•
Emil includes eNB IP addresses and FTM remote test port
Emil includes eNB IP addresses and FTM remote test port
passwords
passwords
•
•
Restrict Emil access only to dedicated personnel
Restrict Emil access only to dedicated personnel
•
•
Emil will remove IMSI and IMEI
Emil will remove IMSI and IMEI
•
•
Should not be a
Should not be a security risk, but must be tested that in
security risk, but must be tested that in
customer network to be working
customer network to be working
•
•
Emil may crash like other Windows applications
Emil may crash like other Windows applications
•
•
To be sure that
To be sure that it is not affecting NetAct performance a
it is not affecting NetAct performance a
separate PC is recomended for tracing
separate PC is recomended for tracing
•
•
Emil Traces include information about site names and
Emil Traces include information about site names and
network quality
network quality
•
•
Traces should be maintained like other confidential
Traces should be maintained like other confidential
information and documents
Emil practical arrangements
Emil practical arrangements
IMSI removal
IMSI removal
UL-DCCH-Message : { UL-DCCH-Message : { message c1 : rrcConnectionSetupComplete : { message c1 : rrcConnectionSetupComplete : { rrc-TransactionIdentifier 2, rrc-TransactionIdentifier 2, criticalExtensions c1 : rrcConnectionSetupComcriticalExtensions c1 : rrcConnectionSetupComplete-r8 : plete-r8 : {{ selectedPLMN-Identity 1, selectedPLMN-Identity 1, dedicatedInfoNAS '07 41 71 08 09 00 00 00 00 00 00 00 03 E0 E0 00 00 1D 02 01 D0 11 27 17 dedicatedInfoNAS '07 41 71 08 09 00 00 00 00 00 00 00 03 E0 E0 00 00 1D 02 01 D0 11 27 17 80 80 21 10 01 00 00 10 81 06 00 00 00 00 83 06 00 00 00 00 00 0A 00 31 80 80 21 10 01 00 00 10 81 06 00 00 00 00 83 06 00 00 00 00 00 0A 00 31 03 E5 C0 04'H 03 E5 C0 04'H }} }} }} Decoded NAS
Decoded NAS message: EMM:ATTACH REQUESTmessage: EMM:ATTACH REQUEST PROTOCOL
PROTOCOL DISCRIMINATODISCRIMINATOR R ----0111 ----0111 EPS EPS MOBILITY MOBILITY MANAGEMMANAGEMENTENT Security
Security header header type type 0000---- 0000---- Not Not security security protectedprotected MESSAGE
MESSAGE TYPE TYPE 01000001 01000001 ATTACH ATTACH REQUESTREQUEST NAS KEY SET IDENTIFIER IE
NAS KEY SET IDENTIFIER IE TCS 0---TCS 0---NAS
NAS key key set set identifier identifier -111---- No -111---- No key key is is availableavailable EPS ATTACH TYPE IE
EPS ATTACH TYPE IE Spare ----0---Spare ----0---EPS
EPS Attach Attach Type Type ---001 ---001 EPS EPS AttachAttach EPS MOBILE IDENTITY IE
EPS MOBILE IDENTITY IE Length
Length 00001000 00001000 88 Identity
Identity digit digit 1 1 0000---- 0000---- 00 Odd/even
Odd/even indication indication ----1--- ----1--- oddodd Type
Type of of identity identity ---001 ---001 IMSIIMSI Identity
Identity digit digit 3 3 0000---- 0000---- 00 Identity
Identity digit digit 2 2 ----0000 ----0000 00 Identity
Identity digit digit 5 5 0000---- 0000---- 00 Identity
Identity digit digit 4 4 ----0000 ----0000 00 Identity
Identity digit digit 7 7 0000---- 0000---- 00 Identity
Identity digit digit 6 6 ----0000 ----0000 00 Identity
Identity digit digit 9 9 0000---- 0000---- 00 Identity
Identity digit digit 8 8 ----0000 ----0000 00 Identity
Identity digit digit 11 11 0000---- 0000---- 00 Identity
Identity digit digit 10 10 ----0000 ----0000 00 Identity
Identity digit digit 13 13 0000---- 0000---- 00
criticality ignore, criticality ignore,
value RRC-Establishment-Cause : mo-Signalling value RRC-Establishment-Cause : mo-Signalling }}
}} }} }}
Decoded NAS
Decoded NAS message: EMM:ATTACH REQUESTmessage: EMM:ATTACH REQUEST PROTOCOL
PROTOCOL DISCRIMINATDISCRIMINAT OR OR ----0111 ----0111 EPS EPS MOBILITY MOBILITY MANAGEMEMANAGEME Security
Security header header type type 0000---- 0000---- Not Not security security protectedprotected MESSAGE
MESSAGE TYPE TYPE 01000001 01000001 ATTACH ATTACH REQUESTREQUEST NAS KEY SET IDENTIFIER IE
NAS KEY SET IDENTIFIER IE TCS 0---TCS 0---NAS
NAS key key set set identifier identifier -111---- No -111---- No key key is is availableavailable EPS ATTACH TYPE IE
EPS ATTACH TYPE IE Spare ----0---Spare ----0---EPS
EPS Attach Attach Type Type ---001 ---001 EPS EPS AttachAttach EPS MOBILE IDENTITY IE
EPS MOBILE IDENTITY IE Length
Length 00001000 00001000 88 Identity
Identity digit digit 1 1 0000---- 0000---- 00 Odd/even
Odd/even indication indication ----1--- ----1--- oddodd Type
Type of of identity identity ---000 ---000 reservedreserved Identity
Identity digit digit 3 3 0000---- 0000---- 00 Identity
Identity digit digit 2 2 ----0000 ----0000 00 Identity
Identity digit digit 5 5 0000---- 0000---- 00 Identity
Identity digit digit 4 4 ----0000 ----0000 00 Identity
Identity digit digit 7 7 0000---- 0000---- 00 Identity
Identity digit digit 6 6 ----0000 ----0000 00 Identity
Identity digit digit 9 9 0000---- 0000---- 00 Identity
Identity digit digit 8 8 ----0000 ----0000 00 Identity
Identity digit digit 11 11 0000---- 0000---- 00 Identity
Identity digit digit 10 10 ----0000 ----0000 00