CLOT KPI Analysis Guideline (RJIL) - v1.0

CLOT KPI Analysis Guideline
(RJIL)
March, 2015
Solution Lab.
Responsible for this document:
, RJIL CLOT KPI Pass/Fail

KPI Fail
ActixOne , AxtixOne
(VoLTE )
,

KPI ActixOne/XCAP/Anyplan-Optimizer
Reference CLOT KPI Fail
Change History:
Changes
Author
Date
Version
Initial Version
Joohyun Kim/Jaewook Jung
29.03.2015
1.0
SAMSUNG Electronics Co., Ltd. Confidential and Proprietary
Table of contents
CLOT KPIs
How to Analyze KPI Items
RSRP
SINR (Info)
Overlapping Servers within 5dB
PCI for all Sectors Observed (Info)
Mac DL Throughput >= 2Mbps
Mac UL Throughput >= 256kbps
Average Mac DL Throughput
Average Mac UL Throughput
Call Drop
TAU Success Rate (Info)
Handover Success Rate
Handover Interruption Time < 100ms
Table of contents
RACH Success Rate

Attach Success Rate
VoLTE Setup Success Rate
Attach Setup Time < 1.5s
IMS Registration Time < 1.5s (Report Only)
VoLTE Setup Time < 1.5s
VoLTE Jitter < 40ms
VoLTE RTP Packet Loss Rate
Average BLER DL (Info)
Average BLER UL (Info)
Appendix
CLOT KPIs
No
KPIs
Call Model
Category
Target
Value
NonExclusion
RSRP
LDL1, LUL
KPI
> 95 %
Dont Consider
SINR >= -2 dB
LDL1, LUL
Info
Dont Consider
Overlapping Servers within 5 dB <= 1
LDL1, LUL
KPI
> 60 %
Dont Consider
Overlapping Servers within 5 dB <= 2
LDL1, LUL
KPI
> 90 %
Dont Consider
PCI for all Sectors Observed
LDL1, LUL
Info
Dont Consider
MAC Throughput DL >= 2 Mbps
LDL
KPI
> 95 %
Dont Consider
MAC Throughput UL >= 256 kbps
LUL
KPI
> 95 %
Dont Consider
Average MAC Throughput DL (kbps)
LDL
KPI
> 16,500
Dont Consider
Average MAC Throughput UL (kbps)
LUL
KPI
> 2,500
Dont Consider
10
Call Drop Rate
LDL1, LUL, VoL
KPI
<1%
Dont Consider
11
TAU Success Rate
LDL1, LUL, VoL
Info
Dont Consider
12
Handover Success Rate
LDL1, LUL, VoL
KPI
> 98 %
Dont Consider
13
Handover Interruption Time < 100ms
LDL1, LUL, VoL
KPI
> 95 %
Dont Consider
14
RACH Success Rate
VoS
KPI
> 99 %
Dont Consider
15
Attach Success Rate
VoS
KPI
> 99 %
Dont Consider
16
VoLTE Setup Success Rate
VoS
KPI
> 99 %
Dont Consider
17
Attach Setup Time < 1.5 s
VoS
KPI
> 95%
Dont Consider
18
IMS Registration Setup Time < 1.5 s
VoS
Info
> 95%
Dont Consider
19
VoLTE Setup Time < 1.5 s
VoS
KPI
> 95%
Dont Consider
20
VoLTE Jitter < 40 ms
VoL
KPI
> 95 %
Dont Consider
21
VoLTE Latency < 100 ms
VoL
KPI
> 95 %
Dont Consider
22
VoLTE RTP Packet Loss Rate
VoL
KPI
<= 1 %
Dont Consider
23
Average BLER DL
LDL1, LUL
Info
Dont Consider
24
Average BLER UL
LDL1, LUL
Info
Dont Consider
Exclusion
1. RSRP
Description
Ratio of RSRP which meet Coverage Objective RSRP Threshold within exclusion zone
Coverage Objective RSRP Threshold value for each sector is referred from Cell Ref File
Sectors with the same clutter category could have different threshold values
BSP for the on-air sites would be only considered as exclusion zone
Clutter Category
DU
DU
DU
DU
DU
MU
MU
MU
MU
MU
RU
RU
RU
RU
SU
SU
SU
SU
SU
SU
Coverage Objective RSRP Threshold

-106.8
-107.4
-108.2
-109.2
-110.5
-108.8
-107.2
-110.2
-110.3
-110.4
-113.6
-114.8
-116.1
-116.2
-111.8
-112.2
-113.2
-113.3
-113.4
-116.3
[Coverage Outage Threshold according to Clutter Category]
[BSP as Coverage Polygon]
1. RSRP
Target Value
Used Call Model
> 95%
LDL1, LUL (Merge)
Formula
(The number of passing RSRP samples / The total number of RSRP samples)
within exclusion zone
Passing RSRP samples : the binned RSRP samples which meet Coverage Objective RSRP Threshold
The total number of RSRP samples : the all binned RSRP samples within exclusion zone
1. RSRP
Analysis using ActixOne and AnyPlan-Optimizer
Check KPI value in ActixOne Report
Find poor RSRP area using ActixOne and AnyPlan-Optimizer

Check RSRP plots with BSP in AnyPlan-Optimizer
Poor RSRP area outside BSP does not have to be considered in the analysis
Find the issue eNBs related with poor RSRP area

Check PCI plots in ActixOne (or AnyPlan-Optimizer)
1. RSRP
Check the coverage of the issue eNBs

Check PCI plots in ActixOne(or AnyPlan-Optimizer)
Execute Sector Swapped Site and Tx Off Site in AnyPlan-Optimizer
Check Sector Plot for each issue eNB in AnyPlan-Optimizer
1. RSRP
Check the issue eNBs alarm status during drive test from LSM
Check the alarms such as RRH/Site/SNMP Down, ECP/UMP/GPS Function Fail , VSWR etc.
I-OR-SUNA-ENB-9005 Alarm History in eNB

No.
Severity
Code
541
Critical
2007
556
Critical
2007
ProbableCause
Alarm time
Clear time
Clear type
Protocol Status Alarm

( Communication Fail : PING )
2015-02-03 12:37
2015-02-03 12:43
Auto

( Communication Fail : PING )
2015-02-03 11:49
2015-02-03 11:49
Auto
10
1. RSRP
Execute AOP in AnyPlan-Optimizer

Fine-tune AOP recommendation manually if necessary
Check the prediction results
RSRP as well SINR and Serving Sector area should be also checked together
Refer to AnyPlan-Optimizer User Guideline for the details

If RSRP KPI is not improved after finishing AOP/Modify Sector Parameter,
it should be checked UEs normality or the measurement environments such as obstacles
Baseline
Prediction
11
1. RSRP
Analysis Results
Antenna Recommendation
Azimuth, M-Tilt and E-Tilt
The issue eNBs lists

All alarms not only for issue eNBs but also for all eNBs in the cluster should be
checked and cleared before drive test
12
2. SINR (Info)
Description
Target Value
Used Call Model
Ratio of SINR that meets SINR Outage Threshold value (-2dB) within exclusion zone
LDL1, LUL (Merge)
Formula
(The number of passing SINR samples / The total number of SINR samples)
Passing SINR sample : the binned SINR samples which meet SINR Outage Threshold value (-2dB)
The total number of SINR samples : the all binned SINR samples within exclusion zone
13
2. SINR (Info)
Analyze poor SINR area

: Analysis methodology would be the similar with RSRP analysis
Refer to RSRP analysis and AnyPlan-Optimizer User Guideline for the details
Find Poor SINR area
Find issue eNBs and check alarms
Execute AOP/Modify Sector Parameters in AnyPlan-Optimizer etc.
ex) removing dominant interference or enhancing the serving signal
Analysis Results
The issue eNB lists

14
2. SINR (Info)
Improve Point
Even through RSRP is good, SINR was very low due to missing neighbor
Check Neighbor relation between two cells
SINR
RSRP
Despite of relatively high RSRP, SINR is

very low due to missing neighbor
15
3. Overlapping Servers within 5 dB
Description
Target Value
> {60% | 90%} for 1 server and 2 servers respectively
Used Call Model
The average number of sectors having RSRP difference with serving best server within 5 dB,
including best server signal within exclusion zone
The number of overlapping sectors is calculated first, then it is averaged by binning
LDL1, LUL (Merge)
Formula
(The number of passing overlapping samples / The total number of bin samples)
Passing overlapping samples : the binned samples which meet the average number of overlapping
sectors within 5 dB <= {1 | 2} within exclusion zone
The total number of binned samples : the all binned samples within exclusion zone
16
Find overlapping area in ActixOne and AnyPlan-Optimizer

Check Serving and Neighbor RSRP in ActixOne (or AnyPlan-Optimizer)
Check Pollution Plots in AnyPlan-Optimizer
If AOP recommendation was already derived, check Pollution Plots in Prediction layer
17
Analyze overlapping area

: Analysis methodology would be the similar with RSRP analysis
Find Overlapping area
ex) removing dominant interference or enhancing the serving signal
Analysis Results
The issue eNB lists

18
Analysis using XCAP
Analyze overlapping area from XCAP

Find issue overlapping area
By XCAP Analyze, try to find the overlapping area
5dB<=2
5dB<=1
19
4. PCI for all Sectors Observed (Info)
Description
Target Value
Used Call Model
The number of observed PCIs during drive test compared with the number of planned PCIs
for the particular cluster
LDL1, LUL (Merge)
Formula
The number of observed PCI during drive test / The number of planned PCIs for the
particular cluster
20
4. PCI for all Sectors Observed (Info)
Check measured PCIs in ActixOne (or AnyPlan-Optimizer)

Check PCI Plots in ActixOne (or AnyPlan-Optimzier)
Check Sector Plot and Display Type function in AnyPlan-Optimizer
Check the un-measured eNBs alarm status during drive test from LSM
Analysis Results
The unmeasured on-air eNB lists

The issue eNB lists
21
5. DL MAC Throughput >= 2Mbps
Description
Target Value
> 95%
Used Call Model
Ratio of summed MAC DL Throughput from device1 and device2 which meets the criteria
(2Mbps) within exclusion zone
LDL1, LDL2 (Separate)
Formula
(The number of passing DL Tput samples (LDL1) + The number of passing DL Tput samples (LDL2)) /
(The total number of DL Tput samples (LDL1) + The total number of DL Tput samples (LDL2))

Passing DL Tput samples : the binned MAC DL Throughput samples which meet 1Mbps within
exclusion zone
The total number of DL Tput samples : the all binned MAC DL Throughput samples within exclusion
zone
22
Analysis using ActixOne, AnyPlan-Optimizer and XCAP
Find poor DL Throughput area using ActixOne and AnyPlan-Optimizer

MAC Layer Throughput Download could be checked in ActixOne report
Only PayloadRate can be displayed in ActixOne
Check DL Throughput plots with BSP in AnyPlan-Optimizer
Poor DL Throughput area outside BSP does not have to be considered in the analysis
23
Find the issue eNBs related with poor DL Throughput area

Check PCI plots in ActixOne (or AnyPlan-Optimizer)
Check the issue eNBs alarm status during drive test from LSM
Refer to RSRP analysis for the details
Modify Antenna Parameters in AnyPlan-Optimzier (Improve DL Edge SINR)

Execute AOP in AnyPlan-Optimzier
Execute Modify Sector Parameters in AnyPlan-Optimzier if necessary
I-OR-SUNA-ENB-9005 Alarm History in eNB

No. Severity Code
541
Critical
556
Critical
ProbableCause
Alarm time Clear time

2015-02-03 2015-02-03
2007 ( Communication Fail :
12:37
12:43
PING )
2015-02-03 2015-02-03
2007 ( Communication Fail :
08:53
11:49
PING )
Clear typ
e
Auto
Auto
24
Check other issues in XCAP

Check missing Neighbors (No HO after multiple MR transmission)
Check Backhaul issues
Server Drop (Error message from FTP server)
Socket Close
Initial attach or IdletoActive fails due to no receiving dlInformationTransfer message
Check FTP Server issues
Analysis Results
The issue eNB lists

Missing NBR list etc.
25
Fail Cases
Poor RF (SINR)
DL MAC Throughput was low due to the low SINR
RF Optimization
DL MAC Throughput fail
26
Fail Cases
Missing Neighbors
DL MAC Throughput was low due to no HO procedure
No HO procedure after multiple MR transmission due to missing Neighbors
RACH procedure by UL Data arrival after multiple MR transmission due to missing
Neighbors
Check Neighbor list in LSM (check missing neighbor)
27
Fail Cases
Interference issue
DL MAC Throughput was low due to UL interference
eNB didnt receive ACK/NACK for FTP DL data on PUSCH due to high IoT in this eNB
Check Interference
LDL PUSCH Tx Power

LDL Power Headroom
LDL BLER
LDL Low DL Throughput

28
Fail Cases
Late Service Request Triggering

UE triggered service request too late after RRC Connection Release, so there happened server drop
Check Backhaul and S1 interface
Server Drop
Release
DL Low Throughput
29
6. UL MAC Throughput >= 256kbps
Description
Target Value
> 95%
Used Call Model
Ratio of MAC UL Throughput which meets the criteria (256kbps) within exclusion zone
LUL
Formula
(The number of passing UL Tput samples / The total number of UL Tput samples)
Passing UL Tput samples : the binned MAC UL Throughput samples which meet 256kbps within
exclusion zone
The total number of UL Tput samples : the all binned MAC UL Throughput samples within exclusion
zone
30
Analyze poor UL Throughput area

: Analysis methodology would be the similar with DL Edge Throughput analysis
Find poor UL Throughput area

Check other issues such as missing neighbor, FTP Server issues in XCAP
BSP
31
Fail Cases
Poor RF (RSRP)
UL MAC Throughput was low due to the low RSRP
RF Optimization
UL MAC Throughput fail
32
Fail Cases
Missing Neighbors
UL MAC Throughput was low due to no HO procedure
After multiple MR transmission, UE didnt start HO procedure due to missing Neighbors
After multiple MR transmission, UE started RACH procedure by UL Data arrival
33
Fail Cases
Backhaul issue
UL MAC Throughput was low due to several backhaul related issues
FTP Server Drop
Socket Close
Small Buffer Size of UE due to small TCP window size of eNB
Late service request after RRC Connection Release
No receiving Security protected NAS message message from MME after RRC connection
setup complete in normal RF environment
Check Backhaul and S1 interface
34
Fail Cases
Inherence issue
UL MAC Throughput was low due to interference(Tx Power : Max, Power Headroom : low, RF
environment : Not bad)
Check interference
UL Throughput
35
7. Average DL MAC Throughput
Description
Target Value
> 16,500kbps
Used Call Model
Summation of the averaged MAL DL Throughput from device 1 and device 2 within
exclusion zone
LDL1, LDL2 (Separate)
Formula
(Average MAC DL Throughput for LDL1 + Average MAC DL Throughput for LDL2) within
exclusion zone
Average MAC DL Throughput : (summation of the binned MAC DL Throughput samples / the total
number of MAC DL Throughput samples) within exclusion zone
36
7. Average DL MAC Throughput
Analyze poor DL Throughput area


37
8. Average UL MAC Throughput
Description
Target Value
> 2,500kbps
Used Call Model
The averaged MAL UL Throughput within exclusion zone
LUL
Formula
Average MAC UL Throughput for LUL within exclusion zone

Average MAC UL Throughput : (summation of the binned MAC UL Throughput samples) / (the total
number of MAC UL Throughput samples) within exclusion zone
38
8. Average UL MAC Throughput
Analyze poor UL Throughput area


BSP
39
9. Call Drop Rate
Description
Target Value
<1%
Used Call Model
Ratio of Dropped VoLTE Calls and Abnormal Release of Data Call
LDL1, LUL and VoL (Separate)
Formula
(The number of VoLTE Call Drops + The number of Abnormal Release for LDL1, LUL and VoL) /
(Summation of the number of Calls Attempts for LDL1, LUL and VoL)
VoLTE Call Drop : Call is terminated by receiving BYE message from network side
Abnormal Release : UE is released without rrcConnectionRelease message
(RRC Connection Request without RRC Connection Release)
The number of Call Attempts for LDL1 : Roundup(Sum(MAC DL Bytes(LDL1))/52,428,800Bytes)
The number of Call Attempts for LUL : Roundup(Sum(MAC UL Bytes(LUL))/10,485,760Bytes)
The number of Call Attempts for VoL : Roundup(Sum(VoLTE Call Duration(VoL))/90sec)
40
9. Call Drop Rate
Analysis using ActixOne and XCAP
Check call drop events in ActixOne

ActixOne does not provide the items which can check VoLTE Call Drop and RRC Drop yet
41
9. Call Drop Rate
Check Call drop events in XCAP

Check the event times for all call drops
Check the detailed messages in XCAP for the fail events
42
9. Call Drop Rate
Fail Cases
Poor RF Environments
UE didnt received RTP packets due to poor RF quality
Check RF environment (RSRP/SINR)
43
9. Call Drop Rate
Fail Cases
Service Reject
UE didnt received RTP packets due to service reject from core network
Check Core network (including IMS Server)
Service Reject
44
9. Call Drop Rate
Fail Cases
UE was released from eNB without rrcConnectionRelease message due to Out-of-Sync.
45
9. Call Drop Rate
Fail Cases
Abnormal release by S1 Interface issue

UE was released from eNB without rrcConnectionRelease message due to S1 interface
Check Backhaul and S1 Interface
No rrcConnectionRelease message
46
10. TAU Success Rate (Info)
Description
Ratio of Tracking Area Update (TAU) Success

TAU procedure : from Tracking Area Update Request to Tracking Area Update Success in NAS L3
message
Target Value
Used Call Model
Formula
The number of TAU Success / The number of TAU Attempts

TAU Success : Receiving Tracking area update accept (DL EPS MM)
TAU Attempt : Sending Tracking area update request (UL EPS MM)
47
Message from XCAP
From Tracking area update request (UL EPS MM) to Tracking area update success (DL EPS MM)
Attempts
Complete
48
Check TAU fail events in ActixOne

ActixOne does not provide the items which can check VoLTE Call Drop and RRC Drop yet
Check TAU fail events in XCAP

Check the specific event time for TAU fails
49
50
Fail Cases
No TAU Update Message

After sending the TAU Message, UE didnt receive the TAU accept
Check Core network (including MME)
TAU Update Fail

51
11. Handover Success Rate
Description
Ratio of Handover Success for the intra-frequency LTE Handover Procedure

Handover procedure : from RRC Connection Reconfiguration (contains HO command)
to LTE MAC RACH Attempt Success for RRC Connection Reconfiguration Complete
Target Value
Used Call Model
>98%
Formula
The number of Handover Success / The number of Handover Attempts

Handover Success : LTE MAC RACH Attempt Success for rrcConnectionReconfigurationComplete
Handover Attempts : Sending rrcConnectionReconfiguration (contains HO command)
52
Message from XCAP
From rrcConnectionReconfiguration message (with HO Command) to LTE MAC RACH

attempt Success
Attempts
Complete
53
Check LTE Handover Fail events in ActixOne
54
Check Handover fail events in Macro and XCAP

Check the specific event time for HO fail
55
Fail Cases
Failure at MSG2
UE didnt receive RAR(MSG2) from eNB due to poor RF environments
56
Fail Cases
RLF during HO
UE didnt succeed Handover due to Radio link failure during HO procedure
Check RF Environments (RSRP/SINR)
57
Fail Cases
PCI Duplication
UE didnt receive RAR(MSG2) due to referring the wrong cell
Check PCI Collision and Confusion in NBR list from LSM
RACH info
SIB2
58
Fail Cases
No Measurement Report
Measurement parameters are not configured in eNB
Check eNB Parameter or configuration
Measurement parameters are not

configured
59
12. Handover Interruption Time < 100ms
Description
Ratio of Handover Success which meets the criteria (100ms) among the intra-frequency LTE
Handover Success Procedure
Handover procedure : from RRC Connection Reconfiguration (contains HO command)
to LTE MAC RACH Attempt Success for RRC Connection Reconfiguration Complete
Target Value
Used Call Model
>95%
Formula
The number of Handover Success within 100ms / The number of Handover Success
Handover Success within 100ms : LTE MAC RACH Attempt Success for
rrcConnectionReconfigurationComplete within 100ms
Handover Success : LTE MAC RACH Attempt Success for rrcConnectionReconfigurationComplete
60
Message from XCAP
From rrcConnectionReconfiguration message (with HO Command) to LTE MAC RACH

attempt Success
Attempts
Complete
61
Check LTE Slow Handover events in ActixOne

Handover Interruption Time fail could be different with LTE Slow Handover events in ActixOne
62
Check Handover Interruption time fail events using Macro

Check the specific event time for HO interruption time fail
63
Fail Cases
Multiple MSG2 re-transmission

UE re-transmitted MSG2 due to poor RF environments
64
13. RACH Success Rate
Description
Ratio of Contention-based RACH Success

RACH Procedure (Contention based): from Random Access Request message(MSG1) to Contention
Resolution message(MSG4)
Target Value
Used Call Model
>99%
VoS
Formula
The number of RACH Success / The number of RACH Attempts(CT Based)

RACH Success : LTE MAC RACH Attempt Success (Success of Message 4)
RACH Attempts(CT Based) :
LTE MAC RACH trigger followed by rrcConnectionRequest
LTE MAC RACH trigger followed by rrcConnectionReestablishmentRequest
65
Message from XCAP
Initial RRC Connection Setup

From LTE MAC RACH trigger by Connection request to LTE MAC RACH attempt Success
Attempts
Complete
RRC Connection Re-establishment

From LTE MAC RACH trigger by Radio Link failure to LTE MAC RACH attempt Success
Attempts
Complete
66
Check Contention-based RACH fail events using Macro and XCAP

Check the event times for RACH fails
RACH triggered by RLF should be also counted
Check the detailed messages in XCAP for the fail event
67
Fail Cases
Failure at MSG2
UE didnt receive Random Access Response (RAR) from eNB due to poor RF environments
RACH will be aborted when T300 timer is expired
68
Fail Cases
Failure at MSG4 due to CT Timer expired

UE didnt receive MSG4 from eNB within the pre-defined CT timer(56sf)
69
14. Attach Success Rate
Description
Ratio of LTE Attach Success

Attach Procedure : from RRC Connection Setup Complete with Attach Request to IMS Registration
Target Value
Used Call Model
>99%
VoS
Formula
The number of Attach Success / The number of Attach Attempts

Attach Success : Receiving SIP 200 OK for IMS Registration after rrcConnectionSetupComplete
with Attach Request
Attach Attempts : Sending rrcConnectionSetupComplete with Attach Request
70
Attach Procedure
71
Message from XCAP
From rrcConnectionSetupComplete message (with Attach Request) to SIP 200 OK message

for IMS REGISTER
Attempts
Complete
72
Check Attach fail events in ActixOne

ActixOne does not provide the items which can check Attach fails yet
73
Check Attach fail events using Macro

Check the event times for attach setup fails
Check the event times for IMS registration fails

Define the issues whether the issue is comes from core network(IMS) or not
Refer IMS issue code in Appendix
74
Fail Cases
Initial Attach Fail by S1 interface issue

UE didnt receive dlInformationTransfer" message after RRC Connection Setup Complete
Check the stability of S1 interface
No message from MME

after rrcConnectionSetupComplete
75
Fail Cases
No Receiving TCP ACK for Sync

UE didnt receive TCP Acknowledgement (SYNC+ACK) for IMS Register after Default Bearer Setup
Check Configuration of N7K router

Check the stability of Backhaul and S1 interface
No receiving ACK message for TCP Sync.
76
Fail Cases
403 Forbidden
UE didnt receive SIP 200 OK message for IMS registration due to IMS server error
Another core side issue codes could be analyzed with the similar way
Check Core network (including IMS server)
No 200 OK for Registration
77
Fail Cases
Initial Attach Fail

After RRC Connection Setup Complete, UE was released due to PDN CONNECTIVITY REJECT
message from MME
Reject Cause : #30 (Request rejected by Serving GW or PDN GW)
Check Core network (including IMS server)
Attach Reject : PDN Connectivity Reject
78
15. VoLTE Setup Success Rate
Description
Ratio of VoLTE Call Setup Success

VoLTE Setup Procedure : from INVITE to 200 OK for INVITE
Target Value
Used Call Model
> 99%
VoS
Formula
The number of VoLTE Setup Successes / The number of VoLTE Setup Attempts
VoLTE Setup Success : Receiving SIP 200 OK message for INVITE
(After a dedicate bearer with QCI 1 is activated)
VoLTE Setup Attempt : Sending SIP INVIVTE message (with originating SDP)
79
VoLTE Setup Procedure
80
Message from XCAP
From SIP INVITE message to SIP 200 OK message for INVITE
Attempts
P-RACK
Complete
200 OK for INVITE
81
Check VoLTE Setup fail events in ActixOne

ActixOne does not implement analyzing function for VoLTE Call in detail
Network Optimization
Call Event Explorer
82
Check VoLTE Setup fail events using Macro

Check the event time for VoLTE setup fails
Long Setup time more than 10 seconds
No Setup time
The number of fails found from Macro file could be different with ActixOne (Refer to Appendix)

Define the issues whether the issue is comes from IMS or not
83
Check the fail events found in ActixOne in XCAP also

Define the issues whether the issue is comes from core network or not
84
Fail Cases
487 Request Terminated

UE didnt receive SIP 200 OK message for invite due to IMS server error
Another core side issue code error could be analyzed with the similar way
No 200 OK for INVITE
85
Fail Cases
500 Server Internal Error

UE didnt receive SIP 183 Session Progress message from IMS due to server error
No 200 OK for INVITE
86
Fail Cases
No 200 OK for PRACK

UE didnt receive SIP 200 OK message for P-RACK when RF environment was good
SIP 200 Status Code : The 200 status code is the ACK message which represents the request was
received and processed successfully
No 200 OK for PRACK
87
Fail Cases
No 100 Trying
UE didnt receive 100 Trying message after invite even through RF Environment is good
88
Fail Cases
No 100 Trying
UE didnt receive 100 Trying message which comes from IMS Server
No 100 Trying between INVITE and PRACK
89
16. Attach Setup Time < 1.5s
Description
Ratio of Attach Setup Success which meets the criteria (1.5s) among the Attach Setup
Success Procedure
Attach Setup Procedure : from RRC Connection Setup Complete with Attach Request to Attach
Complete
Target Value
Used Call Model
>95%
VoS
Formula
The number of Attach Setup Success within 1.5s / The number of Attach Setup Success
Attach Setup Success within 1.5s : Sending Attach complete message (UL EPS MM) after
rrcConnectionSetupComplete with Attach Request
within 1.5s
Attach Setup Success : Sending Attach complete message (UL EPS MM) after
rrcConnectionSetupComplete with Attach Request
90
Attach Setup Procedure
91
Message from XCAP
From rrcConnectionSetupComplete (with Attach Request) to Attach Complete (UL EPS MM)
Attempts
Complete
92
Check Attach setup time fail events in ActixOne

ActixOne does not provide the items which can check Attach setup time fail event yet
93
Check Attach setup time fail events using Macro

Check the event time for attach setup time fails
: Filter out the cases having more than 1.5s

94
Fail Cases
Security protected NAS message Delay

UE received Security protected NAS message from MME too late
Check the stability of S1 Interface

16:59:59.819
16:00:02.088
95
17. IMS Registration Time < 1.5s (Report Only)
Description
Ratio of IMS Registration Success within the criteria (1.5s) among IMS Registration Success
Procedure
IMS Registration Procedure : from PDN Connectivity Request to 200 OK for IMS Registration
Target Value
Used Call Model
>95% (Report Only)
VoS
Formula
The number of IMS Registration within 1.5s / The number of IMS Registration Success
Attach Setup Success within 1.5s : Receiving SIP 200 OK for IMS Registration after PDN Connectivity
Request (UL EPS SM) within 1.5s
IMS Registration Success : Receiving SIP 200 OK for IMS Registration after PDN Connectivity Request
(UL EPS SM)
96
IMS Registration Procedure
97
Message from XCAP
From PDN connectivity request (UL EPS SM) to SIP 200 OK for IMS registration
Attempts
Complete
98
Check IMS registration setup time fail events in ActixOne

ActixOne does not provide the items which can check IMS registration setup time fails yet
99
Fail Cases
504 Server Time-out

UE received 200 OK for IMS registration too late due to server error
Another IMS error code could be analyzed with the similar way
100
18. VoLTE Setup Time < 1.5s
Description
Ratio of VoLTE Setup Success which meets the criteria (1.5s) among the VoLTE Setup Success
Procedure
VoLTE Setup Procedure : from INVITE to 200 OK for IVITE
Target Value
Used Call Model
>95%
VoS
Formula
The number of VoLTE Setup Success within 1.5s / The number of VoLTE Setup Success
VoLTE Setup Success within 1.5s : Receiving SIP 200 OK for INVITE after SIP INVITE within 1.5s
VoLTE Setup Success : Receiving SIP 200 OK for INVITE after SIP INVITE
101
VoLTE Setup Procedure
102
Message from XCAP
From SIP INVITE message to SIP 200 OK message for INVITE
Attempts
P-RACK
Complete
200 OK for INVITE
103
Check VoLTE setup time fail events in ActixOne

ActixOne does not provide the items which can check VoLTE setup time fails yet
104
Check VoLTE setup time fail events using Macro

Check the event time for VoLTE setup time fails
: Filter out the cases having more than 1.5s

105
Fail Cases
QCI1 Activation Delay by IMS

QCI1 Bearer was activated too late

Time: 16:55.25.687
Time: 16:55.31.093
106
Fail Cases
183 Session Progress Delay by IMS

UE received 183 Session Progress message with delay more than 20000ms

Time: 17:25.13.799
Time: 17:25.18.623
107
19. VoLTE Jitter < 40ms
Description
Ratio of VoLTE Jitters which meets the criteria (40ms)

VoLTE Jitters : Statistical variation of the RTP data packet inter-arrival time (RFC 3550)
Target Value
Used Call Model
>95%
VoL
Formula
The number of VoLTE Jitters within 40ms / The number of VoLTE Jitters
Lost VoLTE RTP Packets will be calculated based on the received RTP Packet Sequence Number
108
Check VoLTE Jitter fail events in ActixOne

ActixOne does not provide the items which can check VoLTE Jitter yet
109
Analysis using XCAP
Check Jitter value in XCAP

110
20. VoLTE RTP Packet Loss Rate
Description
Ratio of Lost VoLTE RTP Packet

Causes for packet loss : network congestion, late arrivals etc.
Target Value
Used Call Model
<1%
VoL
Formula
The number of Lost VoLTE RTP Packets / The number of Transmitted VoLTE RTP Packets
Lost VoLTE RTP Packets will be calculated based on the received RTP Packet Sequence Number
111
Check RTP Packet Loss in ActixOne
112
Check RTP Packet Loss events using XCAP

Check the event time for RTP Packet Loss
Packet count/Packet Loss Count
113
Fail Cases
RTP Packets were lost due to poor RF environments(RSRP/SINR)
RF Optimization
114
Packet Loss Point
114
Fail Case
Handover Fail
RTP Packets were lost due to HO Fail
H/O Fail
Packet Loss
115
21. Average BLER DL (Info)
Description
Target Value
Used Call Model
Average NACK rate in DL for the initial transmission within exclusion zone
LDL1, LUL (Merge)
Formula
Average of DL BLER for the binned samples within exclusion zone
116
21. Average BLER DL (Info)
Check BLER DL in XCAP

Check Graph or Table> Qualcomm LTE/LteAdv> PDSCH BLER Info> Total Info> Total BLER
117
22. Average BLER UL (Info)
Description
Target Value
Used Call Model
Average NACK rate in UL for the initial transmission within exclusion zone
LDL1, LUL (Merge)
Formula
Average of UL BLER for the binned samples within exclusion zone
118
Check BLER DL in XCAP

Check Graph or Table> Qualcomm LTE/LteAdv> PDSCH BLER Info> Total Info> Total BLER
119
Fail Cases
High UL BLER due to Interference

High UL BLER was detected withtin specific area due to interference
UL would be affected more severely than DL due to high sensitivity and antenna gain of eNB
Check IoT level in LSM

Detect Interference source using spectrum analyzer at the field
Interference source should be removed before CLOT
High UL BLER area
120
Appendix
CLOT Report Download from ActixOne

CLOT Test Scenarios
Issue Code IMS & EMM
Automation Process
Drive log cycle and Cluster name are required to download CLOT report
Log On to ActixOne(http://49.40.2.69/ActixOne/) with provided login credentials
ID/PW : samsung1/samsung1
122
Follow the below steps sequentially
1. Click on Network
optimization module
2. Click on Task
Management tab
3. Click on filter
tab
4. Select & click

create filter
123
5. Put the filter

Name you want
6. Keep one row for filter ,

select other row and delete
with clicking delete row b
utton
124
7. From Field column sel

ect description and from
operator column select
contains and put Cluster n
ame of required report whi
ch you want to download
in Value column and the
click on Save
125
8. Right click on Task

management window
9. click on Reports
126
10. Double click on R

JIL CLOT Acceptance R
eports template
127
11. Fill the Cluster Name

, Task ID detail and click
on Run
Task ID
Cluster Name
128
12. Save the CLOT report
129
CLOT Test Scenario
Drive Test
It will be used 5 handsets to perform five tests using Accuver DM tool installed in mobile
handset
Each handset will perform one of the test given below
1.
2.
3.
4.
5.
FTP DL 1
FTP DL 2
FTP UL
VoLTE Long
VoLTE Short
All test scenarios are already predefined in Accuver DM tool

Drive test engineer will select different test scenarios per handset
130
Issue Codes
IMS Issue Code for VoLTE
NO
Codes
Description
403
Forbidden
487
Request Terminated
500
Internal Server Error
501
Not Implemented
502
Bad Gateway
503
Service Unavailable
504
Service Time-out
505
SIP Version Not Supported
513
Message Too Large
131
Issue Codes
PDN Connectivity Reject
If the PDN CONNECTIVITY REQUEST cannot be accepted by the network, then MME shall
send a PDN CONNECTIVITY REJECT message to the UE.
This message shall contain an ESM cause value which indicates the reason for rejecting the UE
requested PDN connectivity.
The ESM cause IE typically indicates one of the following ESM cause values;
Refer to section 6.5.1.4 in the 3GPP TS 24.301 for detailed information
#
EMM Cause
EMM Cause
#8
operator determined barring
#35:
PTI already in use
#26
insufficient resources
#38
network failure
#27
missing or unknown APN
#50
PDN type IPv4 only allowed
#28
unknown PDN type
#51
PDN type IPv6 only allowed
#29
user authentication failed
#53:
ESM information not received
#30
request rejected by Serving GW or PDN GW
#54
PDN connection does not exist
#31
request rejected, unspecified
#55
multiple PDN connections for a

given APN not allowed
#32
service option not supported
#95
111: protocol errors
#33
requested service option not subscribed
#112
APN restriction value incompatible

with active EPS bearer context.
#34
service option temporarily out of order
132
Automation Process
Automation Process Improvement
XCAL-Mobile FMS ActixOne

XCAL-Mobile : UE Diagnostic Monitor App. in S4mini, FMS : Fleet Management System, ActixOne : Automatic field test log analysis and report
generation tool
XCALMobile
Content
Collect Log file
File Type
DRX
Content
Management Call Scenario, IMEI & Log file
File Type
Convert DRX DRM
Storage
600GB Upgrade 6TB (10DEC ~ 15DEC2014)
Processing
Processing 1 = max 2,000 Devices
Save file
The type of Log file is DRX saved 3 month (User option)
FMS
DRX
DRM
(Storage 1 / Processing 1)
(Storage 1 / Processing 28)
B
DRX
Backup
Server
Point
ActixOne
Content
Save Log files
File Type
DRX
Storage
10TB
Save file
The type of Log file is DRX saved

(Not delete)
Content
Generation SCFT & CLOT

report
Provide Plot & Map
File Type
DRM
Storage
7TB upgraded 60TB

(15JAN2015)
Processing
sever
8 28 upgraded
1 = 16 files processing
Improvement Plan
Target
Owner
To add file check and feedback process in FMS when engineer upload log files.
End of Feb.
Accuver
B-1
To enhance link bandwidth between FMS and ActixOne to raise file transfer speed up.
ASAP
RJIL
B-2
SAMSUNG
Electronics
Co.,
Ltd. Confidential and Proprietary TBD
Need to add status managing system
between
FMS and
ActixOne.
Accuver, RJIL
133
Copyright and Confidentiality

Copyright 2014 SAMSUNG Electronics Co., Ltd. SAMSUNG Electronics reserves the right to make changes to the specifications
of the products detailed in this document at any time without notice and obligation to notify any person of such changes.
Information in this document is proprietary to SAMSUNG Electronics Co., Ltd. No information contained here may be copied,
translated, transcribed or duplicated by any form without the prior written consent of SAMSUNG Electronics.

CLOT KPI Analysis Guideline (RJIL) - v1.0

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

CLOT KPI Analysis Guideline (RJIL) - v1.0

Enviado por

Direitos autorais:

Formatos disponíveis

CLOT KPI Analysis Guideline

Responsible for this document:

, RJIL CLOT KPI Pass/Fail

Joohyun Kim/Jaewook Jung

SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

RACH Success Rate

Overlapping Servers within 5 dB <= 1

Overlapping Servers within 5 dB <= 2

PCI for all Sectors Observed

MAC Throughput DL >= 2 Mbps

MAC Throughput UL >= 256 kbps

Average MAC Throughput DL (kbps)

Average MAC Throughput UL (kbps)

Call Drop Rate

LDL1, LUL, VoL

TAU Success Rate

LDL1, LUL, VoL

Handover Success Rate

LDL1, LUL, VoL

Handover Interruption Time < 100ms

LDL1, LUL, VoL

RACH Success Rate

Attach Success Rate

VoLTE Setup Success Rate

Attach Setup Time < 1.5 s

IMS Registration Setup Time < 1.5 s

VoLTE Setup Time < 1.5 s

VoLTE Jitter < 40 ms

VoLTE Latency < 100 ms

VoLTE RTP Packet Loss Rate

SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

How to Analyze KPI Items

Coverage Objective RSRP Threshold

[Coverage Outage Threshold according to Clutter Category]

[BSP as Coverage Polygon]

SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

Used Call Model

LDL1, LUL (Merge)

SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

Analysis using ActixOne and AnyPlan-Optimizer

Check KPI value in ActixOne Report

Find poor RSRP area using ActixOne and AnyPlan-Optimizer

Find the issue eNBs related with poor RSRP area

SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

Analysis using ActixOne and AnyPlan-Optimizer

Check the coverage of the issue eNBs

SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

Analysis using ActixOne and AnyPlan-Optimizer

I-OR-SUNA-ENB-9005 Alarm History in eNB

Protocol Status Alarm

Protocol Status Alarm

SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

Analysis using ActixOne and AnyPlan-Optimizer

Execute AOP in AnyPlan-Optimizer

Refer to AnyPlan-Optimizer User Guideline for the details

SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

The issue eNBs lists

SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

Used Call Model

LDL1, LUL (Merge)

SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

Analysis using ActixOne and AnyPlan-Optimizer

Check KPI value in ActixOne Report