KPI Investigation Methodologies

Processes, methods and tools to troubleshoot

Agenda

1. Introduction 2. KPI Investigation Methodology Overview 3. First Level Investigation 4. Deep Investigation

Introduction

3 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

Introduction
KPI degradation after Swap/Software Upgrade / Hardware Replacement / Feature Activation is a sensitive issue for customers. KPI degradations need to be quickly detected through accurate monitoring.

Main objectives of this document:

Present the recommended Processes,and Tools to troubleshoot efficiently the performance issues observed during Global Market. In particular, clarify the roles of the on-site team and back-office teams during the First Level Investigation and Deep Investigation steps of the investigation

4 | UA7KPI Investigation Methodology

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

KPI Investigation Methodology Overview

5 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

KPI Investigation Methodology Overview

Investigation process (organizational view) Monitor the network (customer) KPI degradation Monitor the network (ALU)

First Level Analysis

First Level Investigation

Alarms,Configuration

Deep Investigation

Open an AR that should include all the outputs from the First Level Analysis Optimization/Suppo rt/GPS troubleshooting experts

- Analyze the issue based on all available data (including traces) - Maintain a comprehensive document (usually Powerpoint) with investigation results and current action plan Deeply analyze parameter settings and counters subject matter experts Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction. (GPS group)

- Collect additional traces (RNC and NodeB

internal traces, Network Analyzer traces, CTg, Call Failure Trace, etc.)

6 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

- Change parameters and perform the related monitoring - Perform drive tests - Apply fixes and assess performance recovery

KPI Investigation Methodology Overview Investigation process (functional view)

Customer feedback KPI degradation Daily monitoring

Check parameter changes

Basic counter analysis to identify the TopN most impacted NodeBs or Cells

Check alarms & stability

Record predefined set of Call Trace & product traces

First Level Investigation

- Logs are uploaded as recommended, then AR is opened - Before considering going deeper (e.g. open CR, trigger more support), check the Release Notes applicable to current SW load to verify if issue is a known issue

CS CDR Troubleshooting PS R99 CDR Troubleshooting HSxPA CDR Troubleshooting 7 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

Throughput Troubleshooting Accessibility Troubleshooting Mobility Alcatel-Lucent Internal Troubleshooting Proprietary Use pursuant to Company instruction.

Deep Investigation
Optimization/Support/ GPS and troubleshooting experts

KPI Investigation Methodology Overview

Weekly performance investigation meetings

For each identified performance issue , technical meetings ;these meetings are usually chaired by a single matter expert or Optimization troubleshooting expert should be scheduled on a weekly basis in order to: Share the latest investigation results from each involved team Synch-up on the recent key events and data logs Agree on potential new axis of investigations Determine a clear action plan (next steps) and assess its feasibility

8 | UA7.1.3 KPI Investigation Methodology

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

KPI Investigation Methodology Overview

Useful Links Internal tools (ALU-proprietary) [1/3]

WPS (Wireless Provisioning System):

1. 2.

https://wcdma-ll.app.alcatel-lucent.com/livelink/livelink.exe?func=ll&objId=63414868 Launch WPS installation (file wips-Vxxx_windows.exe) When prompted during installation, load the relevant UMTS plugin (file UMTS_Access-Vxxx.wipsar)

Call Failure Trace:

https://wcdma-ll.app.alcatel-lucent.com/livelink/livelink.exe?func=ll&objId=57344700 At above link can be found: Presentations on CFT capabilities and CFT usage examples CFT Reader tool

eDAT (Evolved Data Analysis Tool):

RNC Log Collection:

http://navigator.web.alcatel-lucent.com/index.htm https://wcdma-ll.app.alcatel-lucent.com/livelink/livelink.exe?func=ll&objId=61064195 https://wcdma-ll.app.alcatel-lucent.com/livelink/livelink.exe?func=ll&objId=60312775

NodeB Log Collection:

9 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

KPI Investigation Methodology Overview

Useful Links Internal tools (ALU-proprietary) [3/3]

RFO Product:
http://aww.srd.alcatel.com/icr/SDCT/sdct.php The License must be requested through the SDCT (Software Delivery Center Tool) homepage accessible at above link. Once under the SDCT homepage:

1.

Download the SDCT User Guide (Online documentation PowerPoint link at the top of the page)
and read it

2.

Download the W-CDMA RFO License Ordering presentation (W-CDMA_RFO_License link at the bottom-right of the page) and read it

3.

Click on the Request link at the middle of the page to initiate the license request procedure. - If you are not yet registered in the SDC database, you will first need to register (Registering button) - When prompted the template to select is RFO Product license

10 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

KPI Investigation Methodology Overview

Useful Links External tools

Wireshark:
Wireshark: http://www.wireshark.org/download.html WinPcap: http://www.winpcap.org/archive/4.1beta5_WinPcap.exe
- Packet capture and filtering engine of many open source and commercial network tools - Used by Wireshark

Iperf:
https://wcdma-ll.app.alcatel-lucent.com/livelink/livelink.exe?func=ll&objid=57980928

11 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

First Level Investigation

12 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

First Level Investigation

Pre-Requisites: Collection of reference data before Upgrade [2/2] The KPI Investigation Methodology is fully applicable only if the following reference data are properly collected in case of Swap/the SW Upgrade / HW Replacement / Feature Activation (continued): Snapshot (.xml or .xcm file containing the parameter configuration data)
Daily network Snapshots should be collected during at least 1 week prior to the upgrade, to check configuration (parameters) stability. The Snapshots should be stored in a LiveLink repository for easy access by back-office teams.

Hardware Inventory
HW Inventory files should be collected.

Drive Test. Call Trace.

13 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

First Level Investigation

Pre-requisites:

Tool X: Internal tool (ALU-proprietary) Tool Y: External tool

The KPI Investigation Methodology refers to many internal (ALU-proprietary) and external troubleshooting tools

Tools needed for First Level Investigation WPS (Wireless Provisioning System)

Tools needed for Deep Investigation

(i.e. to collect additional logs and deeply analyze the issue)

NPO (Nw Performance Optimizer)

UTRAN Call Trace Wizard HFB (Historical Fault Browser) RNC Log Collection NodeB Log Collection BTS Set Traces

eDAT, to post-process UE traces and Call Traces RFO Product, to post-process Call Traces WQA, to post-process CTn and CFT traces CFT Reader, to post-process CFT traces Remote Connection to NodeB (through Telnet session or by using TIL tool), to check the NodeB status and collect

specific NodeB internal traces (e.g. IMT logs) Audit Tool, to produce the Audit and NodeB Profiling excel files based on Snapshot and HW Inventory Wireshark, Windump, Iperf, DU Meter to investigate E2E Tput

14 | UA7.1.3 KPI Investigation Methodology

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

First Level Investigation

Selection of TopN most impacted network elements Customer feedback KPI degradation Daily monitoring

Check parameter changes

1/ Basic counter analysis to identify the TopN most impacted NodeBs or Cells

Check alarms & stability

- Focus on the TopN NEs determined through counter analysis - Use e.g. HFB tool

For this, use standard NPO Views 2/ Also try to roughly identify the failure causes

Record pre-defined set of Call Trace & product traces

- Launch CTg+CFT and RNC Log Collection on the TopN NEs - Launch NodeB Log Collection and BTS Set Traces with NodeB KPI AP template on the Top3 NodeBs

- Logs are uploaded as recommended, then AR is opened - Before considering going deeper, check the Release Notes applicable to current SW load to verify if issue is a known issue

First Level Investigation

Deep Investigation
Rule to select the TopN most impacted (or worst) network elements:
15 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

If degradation is observed at RNC level Identify the TopN NodeBs If degradation is observed at NodeB level Identify the TopN Cells
Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

First Level Investigation

Parameter check [2/2] Method to compute all the differences between 2 Snapshots with WPS tool: Import in WPS the earliest available Snapshot after the observed KPI degradation. Use Replace initial snapshot and discard existing workorders option.

Under the same WPS application, import on top of previously loaded Snapshot the latest available Snapshot before the observed KPI degradation. Use Resynchronize the planned configuration with new snapshot option.

Under the Workorders tab you will see the delta workorder to transition from old to new Snapshot, showing all the differences. You can analyze the computed differences here, or make a right-click on the table to export them as an .html file and analyze them under Excel (recommended option).
16 | UA7.1.3 KPI Investigation Methodology | Feb. 2011 Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

First Level Investigation

Call Trace [1/2] CTB/CTg Call Trace logs:
In order to collect the relevant data in CTg logs while capturing a sufficiently high number of failures, the CTg session can be configured to capture only a few specific call types.

CFT Call Trace logs:

Since UA7.1 release, Call Failure Trace (CFT) logs have proven to be a powerful additional type of Call Trace for KPI investigation. A CFT record is being captured each time an RRC Connection failure, RAB Assignment failure or RAB Drop occurs. Differently from CTg logs, CFT logs do not contain the whole call flow but are basically a detailed snapshot of the call at failure time; CFT logs are typically smaller than CTg logs. In typical live environment a CFT session will capture the great majority of call failures, whereas a CTg session usually captures only a relatively small part of the total number calls. A CFT session will attempt to capture all call failures, regardless to the call types specified in the CTg Call Trace template CFT is particularly powerful for Multi-RAB types of KPI degradations

CFT logs should be collected each time a CTg session is launched. This can be done by making sure to use a CTg Call Trace template that includes the record called UeCallFailureFull. CFT logs should always be made available together with the CTg logs under the log storage location (e.g. in a folder named CallTrace).

17 | UA7.1.3 KPI Investigation Methodology

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

First Level Investigation WPS snapshot delta (e.g. .html format), 2 snapshots (latest snapshot before the degradation, earliest snapshot after the degr.) in folder Snapshots Output of RNC Log Collection in folder RNC logs Output of NodeB Log Collection (at least 1 BTS) in folder NodeB logs CTg traces in folder CallTrace Call Failure Trace data in folder CallTrace

18 | UA7.1.3 KPI Investigation Methodology |

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

Deep Investigation

19 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

Deep Investigation
Overview Customer feedback KPI degradation Daily monitoring

First Level Analysis

First Level Investigation

- Before considering going deeper, check the Release Notes applicable to current SW load to verify if issue is a known issue.Alarms and Configuration.

Deep Investigation
CS CDR Troubleshooting
PS R99 CDR Troubleshooting HSxPA CDR Troubleshooting
20 | UA7.1.3 KPI Investigation Methodology

Throughput Troubleshooting
Accessibility Troubleshooting Mobility Troubleshooting
Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

Optimization/Support/ GPS troubleshooting experts.

Deep Investigation
Counter-based in-depth analysis

Deeper counter-based analysis is needed in order to characterize the degradation. The aim is: To narrow down as much as possible where the degradation occurs (product, board, procedure) To identify a common profile of TopN most impacted cells (HW, configuration, traffic profile, etc) The analysis is done with:

Counter correlation => attempt to correlate the general counter showing the degradation and a counter for a specific lower level procedure.
Comparison of distribution of failure types (e.g. which cause of drop has increased the most)

Counters-based analysis is also used:

To quantify patterns identified by CTg analysis To assess the KPI recovery after fix delivery
21 | UA7.1.3 KPI Investigation Methodology 1 Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

Deep Investigation
CS CDR Troubleshooting cookbook CS CDR troubleshooting Analysis of relevant counters for CS CDR degradation

Analysis of CFT data

Analysis of CTg traces

Launch CTg+CFT session on TopN cells

Consolidate CTg + CFT + NPO analysis (cause of drops, drop scenario, IMSI)

Consolidate global analysis (using all tools outputs)

22 | UA7.1.3 KPI Investigation Methodology

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

Deep Investigation
PS R99 Troubleshooting cookbook PS R99 CDR troubleshooting Analysis of relevant counters for PS CDR degradation

Analysis of CFT data

Analysis of CTg traces

Launch CTg+CFT session on TopN cells

Consolidate CTg + CFT + NPO analysis (cause of drops, drop scenario, IMSI)

Consolidate global analysis (using all tools outputs)

23 | UA7.1.3 KPI Investigation Methodology |

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

Deep Investigation
HSxPA CDR Troubleshooting cookbook HSxPA CDR troubleshooting Analysis of relevant counters for HSxPA CDR degradation Analysis of CFT data Analysis of CTg traces Launch CTg+CFT session on TopN cells

Consolidate CTg + CFT + NPO analysis (cause of drops, drop scenario, IMSI)

If needed, perform Drive tests + UE traces Consolidate global analysis (using all tools outputs)

24 | UA7.1.3 KPI Investigation Methodology

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

Deep Investigation
Mobility troubleshooting cookbook Mobility troubleshooting Analysis of relevant counters for Mobility degradation Launch CTn session on TopN cells

Analysis of CTg traces

Consolidate CTg + NPO analysis Perform WQA CTn analysis If needed, perform Drive tests + UE traces Consolidate global analysis (using all tools outputs)

25 | UA7.1.3 KPI Investigation Methodology

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

Deep Investigation
Accessibility troubleshooting cookbook Accessibility troubleshooting Analysis of relevant counters for Accessibility degradation Analysis of CFT data Analysis of CTg traces Launch CTg+CFT session on TopN cells

Consolidate CTg + CFT + NPO analysis In case of RRC degradation: Perform drive-tests (UE traces) Consolidate global analysis (using all tools outputs)

26 | UA7.1.3 KPI Investigation Methodology

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.

www.alcatel-lucent.com www.alcatel-lucent.com

27 | UA7.1.3 KPI Investigation Methodology | Feb. 2011

Alcatel-Lucent Internal Proprietary Use pursuant to Company instruction.