New AR:

View AR | AR without Proprietary

History | States | Assignment | Timetracking
AR Text Search | More | CARES Home

Assistance Request 1-5532074

Customer Ticket N/A

Short

Repeated TMUs reset on RNC231

Description

Current

Fix in LR13.3, LR14.2, LR14.2.5, LR15.3 loads

Summary

State

Closed : Solution/Service Provided

Outage

No

Severity

Priority

Service Request Remote Technical Support

Request Type

Defect

Sub-Type

Software

Category

Software

Sub-Category

Software Defect

Internal

Yes

Assignment Events
Product
Product

9370 RNC

Version

LR13.3.W

No Change Request is linked to this AR

Solution Source 1400075
Product Location
Site

ETISALAT ABU DHABI

Instance

9370RNC-Etisalat : In Service

Site Company

Emirates Telecommunications Corporation

City

Abu Dhabi

Country

United Arab Emirates

Contact
Name

Karim IBRAHIM ABD EL NABY

Company

Alcatel-Lucent : Open on behalf of Customer

Phone
20 127 109 0616
Karim.Ibrahim@alcatel-lucent.com
Request Method

Email

Email-Eng

Dates
Occurred
(GMT+4)

12-Jan-2015 21:27

Time now

29-Aug-2015 13:08

Reported

12-Jan-2015 21:27

AR Created

12-Jan-2015 21:30

Service Start

12-Jan-2015 21:27
Next Customer Contact

29-Jan-2016 03:00

Responded
12-Jan-2015 21:53
SA - Calculated

Respond Target

12-Jan-2015 21:57

Restored
15-Jan-2015 17:00
SA - Calculated

Restore Target

15-Jan-2015 17:14

Resolved
12-Mar-2015 11:33
SA - Calculated

Resolve Target

28-Apr-2015 15:01

Targets for Restore & Resolve were extended by 45 days

to account for Pending time
Closed

12-Mar-2015 11:35

Last Modified

10-Jul-2015 16:02

Modified By

archive

Entitlement
Agreement

242895 (OXIA 627085)

Covered Service TS 24x7 (Gold, Wireless Mobile Access)

Script

This request is entitled to service.

People
Owner

TSCr-MoA-WCDMA-SK : rories

Assignee

TSCr-MoA-WCDMA-SK : rories

Copy To

rories

Referred 1

TEC-MoA-WCDMA_RNC-FR : remiburie

Resolve Group

TEC-MoA-WCDMA_RNC-FR

Submitter

kmajda

Description
Attachments

[empty]

Showing Investigation and Proprietary logs together in time order

separately

Show

Investigation Log and Proprietary Log

1. 12-Jan-2015 21:30
kmajda

Update to Current Summary: TEC support to check the Repeated TMUs reset on
RNC231,
After investigation on the AR 1-5531112 its recommended to open another
ticket
with this description,

2. 12-Jan-2015 21:53
rories

From: RIES, Robert (Robert)

Sent: Monday, January 12, 2015 6:50 PM
To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **
Cc: BERIDY, AHMED (AHMED); BOSLEY, Tim (Tim)** CTR **; KHEDR, MAHMOUD
(MAHMOUD);
REDA, RAMY (RAMY); ABDEL-HALIM, SAYED (SAYED); Berky, Dusan (Dusan);
MOHAMED,
GEHAD (GEHAD); Kosorin, Rastislav (Rastislav); EL-MIDANY, AHMED (AHMED);
OKASHA,
TAHER (TAHER)
Subject: RE: Etisalat UAE/Repeated TMUs reset on RNC231 /1-5532074

Hi Karim

Based on our just finished conference call, you opened this ticket to
support
deeper investigation of TMU issue on RNC231.
For your information - basic logs were collected so we started the
investigation
procedure already. In case of another info, I will page you if needed.

Best Regards

Robert Ries
Update to Current Summary: TPS w.i.p

3. 13-Jan-2015 17:39
rories

Update to Current Summary: Escalated to TPS L3

4. 13-Jan-2015 17:39
rories

ALCATEL-LUCENT PROPRIETARY

Dear TEC team

This ticket is opened based on new issue founded during outage in ETC (UAE)
network, which was solved in another AR 1-5531112.

Issue description:
During emergency recovery activity connected with almost all TMU pendings on
RNC231 (investigated in AR 1-5531112), we found out in hfb the huge amount
of
alarms like this:

RNC231;NW/UTRAN RNC/RNC231 RNCEquipment/0 CNode/0;12-10-2014 14:21:18;12-102014

14:42:42;quality of
service;warning;cleared;softwareProgramAbnormallyTerminated;>
Local value : ANO_RESTART_CONTEXT on PSFP slot 13 Processor role First PMCTMU;>
Local value : ANO_RESTART_CONTEXT on PSFP slot 13 Processor role First PMCTMU
Specific Problem:> Local value : ANO_RESTART_CONTEXT on PSFP slot 13
Processor
role First PMC-TMU, manu:[unavailable value], plmno:Ns-FailCause,
common:[unavailable value],
AdditionalInformationFromNE:additionalInformation = {
common = { information = { postMortem = fmNormal eqp3GLocation = {

hardSlotPosition = { shelfNumber = 0 positionInShelf = 13 } hardconf = { } }

} }
plmno = { information = { plmnoRestartContext = { boardType = v-pmcboardType
cause = v-ns-FailCause overFMTreshold = v-fmtreshold-notExceeded } } } manu
= {
information = "FM PMDZ is written and valid BaseOS PMDZ is empty. " "FM nb
event
0. FM threshold has NOT occured. "

} } ;RNC_2028_30467;

Alarms started the 10th Dec 2014 and were connected with link Iur/8, which
was
therefore blocked (in order to stop TMU resets).
During outage investigation there were many alarms like this:

Lp/9 Ap/1; 2015-01-12 15:05:41.92

MSG minor equipment softwareError
00009000
ADMIN: unlocked

OPER: enabled

USAGE: active

AVAIL:

PROC: reporting

CNTRL:

ALARM:

STBY: notSet

UNKNW: false

Id: 0900000B

Rel: Shelf Card/9

Com: TMU -- EXCEPTION:Memory Manager error[memPartFree] at block 0x4fee0

Prev Hdr [0xffffffff]
[0xffffffff]

Curr Hdr [0x4fed0]

both start and end addr is invalid.

both start and end addr is invalid.

Task:0x7d570000: UeCallp
DAR: 0x00000000
PC: 0x003a94f0 taskUnlock +0x174

Next Hdr

LR: 0x6af9656c
_Z63ue_call_in_reloc_AWAIT_DRNC_TO_SRNC_READY_Ue_drnc_to_srnc_readyP23
CPI_UeDrncToSrncReady_tP16Uecall_
Int: 0/0/0/114; /clearlib/vobs/I-Node/sw/PS/Base/PsHAL/psFltSwr/src/ps
FaultHandling.cc; 2925; PATCH: RID37300E1953

From ppc file we can see a lot of:

Lp/5 Ap/1 ; 2014-12-30 08:24:30.53

MSG minor

equipment

00009000

ADMIN: unlocked

OPER: enabled

USAGE: active

AVAIL:

PROC: reporting

CNTRL:

ALARM:

STBY: notSet

UNKNW: false

Id: 5000310
Com: TMU -- EXCEPTION:Memory Manager error[memPartFree] at block 0x4fee0
Prev Hdr [0xffffffff]

Curr Hdr [0x4fed0]

Next Hdr [0xffffffff]

both start and end addr is invalid.

both start and end addr is invalid.

We found also another ticket with similar symptons, maybe which can be
useful in
your investigation - AR 1-5498787

Please can you ivestigate the probem and tell us:

a) What is a root cause of the problem ?
b) Fix delivery plan for this issue.

Appropriate logs are on CUBA server 135.120.182.10:

either in directory: /traces/OUTAGE_RCA_ER_GPS/1-5531112

or in directory: /traces/Etisatat UAE/1-5532074 Repeated TMUs reset on
RNC231

Thank you
Best Regards
Robert Ries

5. 13-Jan-2015 19:14
remiburie

ALCATEL-LUCENT PROPRIETARY

Hi,

CR 1434789 has been ceated and escalated to design.

Regards,
Rmi

6. 14-Jan-2015 11:35
rories

From: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Sent: Monday, January 12, 2015 7:04 PM
To: RIES, Robert (Robert)
Cc: BERIDY, AHMED (AHMED); BOSLEY, Tim (Tim)** CTR **; KHEDR, MAHMOUD
(MAHMOUD);
REDA, RAMY (RAMY); ABDEL-HALIM, SAYED (SAYED); Berky, Dusan (Dusan);
MOHAMED,
GEHAD (GEHAD); Kosorin, Rastislav (Rastislav); EL-MIDANY, AHMED (AHMED);
OKASHA,

TAHER (TAHER)
Subject: RE: Etisalat UAE/Repeated TMUs reset on RNC231 /1-5532074

Hi Reis,

Attached a network snapshot as requested,

Action taken today:

-

IP transport team action about 11 am.

TMU reset observed with a very high trend with zero traffic on the
RNC231 as
below,
LP switchover action taken to try solve the problem ' Issue enhanced
for 10 min
and back to the same behavior.
Recommended
shelf reset'

by TSO (Reis) after the ticket opened to perform a RNC

issue enhanced, RNC carrying very low traffic, Alarms still exist.
TEC invitagate the issue and find that the issue is the IUR towards
RNC206.
-

IUR link locked, Alarms cleared, unlocked again the alarms back.

Proposed to lock the IUR link,

As discussed we have the same IUR link towards RNC 206 working fine with
RNC235.

Thanks,

-BR,

Karim

7. 26-Jan-2015 12:34
remiburie

ALCATEL-LUCENT PROPRIETARY

Hi,

CR

1434789 is a duplicate of

CR1400075 delivered in LR14, it is planned to

backprop this fix to next LR13 MNCL.

Regards,
Rmi

8. 26-Jan-2015 13:45
rories

From: RIES, Robert (Robert)

Sent: Monday, January 26, 2015 10:22 AM
To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **
Cc: BERIDY, AHMED (AHMED); BOSLEY, Tim (Tim)** CTR **; KHEDR, MAHMOUD
(MAHMOUD);
REDA, RAMY (RAMY); ABDEL-HALIM, SAYED (SAYED); MOHAMED, GEHAD (GEHAD); ELMIDANY,
AHMED (AHMED); OKASHA, TAHER (TAHER)
Subject: RE: Etisalat UAE/Repeated TMUs reset on RNC231 /1-5532074

Hi Karim

Preliminary answer from design:

CR 1434789 opened for this AR is a duplicate of CR 1400075 already delivered

to
LR14.
There is planned back propagation to the next LR13 MNCL.

Best Regards

Robert Ries

9. 13-Feb-2015 18:22
remiburie

ALCATEL-LUCENT PROPRIETARY

Hi,

Design propose as workaround to deactivate feature PM28436.3

isEnhancedIurRabConfigEnabled to be set to false.
The faulty SRB will be refused but possibility also to refuse RAB that RNC
was
able to manage with this feature enabled.

Regards,
Rmi

10. 16-Feb-2015 12:46

rories

From: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Sent: Tuesday, February 03, 2015 8:25 AM
To: RIES, Robert (Robert)

Cc: BERIDY, AHMED (AHMED); BOSLEY, Tim (Tim)** CTR **; KHEDR, MAHMOUD
(MAHMOUD);
REDA, RAMY (RAMY); ABDEL-HALIM, SAYED (SAYED); MOHAMED, GEHAD (GEHAD); ELMIDANY,
AHMED (AHMED); OKASHA, TAHER (TAHER)
Subject: RE: Etisalat UAE/Repeated TMUs reset on RNC231 /1-5532074

Hi Reis,

You close this AR, and please when the CR is ready share it with us.

Thanks,

-BR,
Karim

11. 16-Feb-2015 12:47

rories

From: RIES, Robert (Robert)

Sent: Monday, February 16, 2015 9:46 AM
To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **
Cc: BERIDY, AHMED (AHMED); BOSLEY, Tim (Tim)** CTR **; KHEDR, MAHMOUD
(MAHMOUD);
REDA, RAMY (RAMY); ABDEL-HALIM, SAYED (SAYED); MOHAMED, GEHAD (GEHAD); ELMIDANY,
AHMED (AHMED); OKASHA, TAHER (TAHER)
Subject: RE: Etisalat UAE/Repeated TMUs reset on RNC231 /1-5532074

Hi Karim

Design proposed next workaround:

Please deactivate the feature PM28436.3 "isEnhancedIurRabConfigEnabled" - it

needs
to be set to false.
By this WA the RNC will refuse RABs with faulty SRB. Once there will be fix
(based
on CR 1434789 & 1400075) available in LR13 MNCL, I let you know.

Thanks
Best Regards

Robert Ries

12. 16-Feb-2015 12:47

rories

ALCATEL-LUCENT PROPRIETARY

De : RIES, Robert (Robert)

Envoy : lundi 16 fvrier 2015 08:37
: BURIE, REMI (REMI)
Objet : FW: 1-5532074 has been modified

Hi Rmi

Few questions:

Regarding the second sentence in your update - you wrote, that by

deactivating of

feature PM28436.3 there will be refused RABs with faulty SRB. OK, but once
there
will be available fix in LR13 MNCL, the feature PM28436.3 could be activated
back
?
Regarding CR 1434789 (or CR 1400075) - customer agreed to close this AR in
fact,
but he needs to be alerted once fix is available in LR13 MNCL (his mail in
attachment). Do you know when it will be ? Because in case it will be in
short
time, I let the ticket open to wait for fix availability.

Thanks
Best Regards

Robert Ries

13. 16-Feb-2015 12:48

rories

ALCATEL-LUCENT PROPRIETARY

From: BURIE, REMI (REMI)

Sent: Monday, February 16, 2015 9:27 AM
To: RIES, Robert (Robert)
Subject: RE: 1-5532074 has been modified

Hi Robert,

Design is working on it now, I don't have a date for the fix already.
If I have one I will communicate it of course.

Regards,

Rmi BURIE

14. 26-Feb-2015 12:51

rories

From: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Sent: Monday, February 16, 2015 1:52 PM
To: RIES, Robert (Robert)
Cc: BERIDY, AHMED (AHMED); BOSLEY, Tim (Tim)** CTR **; KHEDR, MAHMOUD
(MAHMOUD);
REDA, RAMY (RAMY); ABDEL-HALIM, SAYED (SAYED); MOHAMED, GEHAD (GEHAD); ELMIDANY,
AHMED (AHMED); OKASHA, TAHER (TAHER); MERAD, ABDESSAMAD (ABDESSAMAD);
BERIDY,
AHMED (AHMED)
Subject: RE: Etisalat UAE/Repeated TMUs reset on RNC231 /1-5532074

Hello Reis,

Thanks for your feedback, Please what will be the impact the IUR behavior
after
unlock, if this deactivation will enhance the KPIs? Or it's only related to
TMU
reset issue?

Thanks,

-BR,

Karim

15. 26-Feb-2015 12:52

rories

From: RIES, Robert (Robert)

Sent: Tuesday, February 17, 2015 8:36 AM
To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **
Cc: BERIDY, AHMED (AHMED); BOSLEY, Tim (Tim)** CTR **; KHEDR, MAHMOUD
(MAHMOUD);
REDA, RAMY (RAMY); ABDEL-HALIM, SAYED (SAYED); MOHAMED, GEHAD (GEHAD); ELMIDANY,
AHMED (AHMED); OKASHA, TAHER (TAHER); MERAD, ABDESSAMAD (ABDESSAMAD);
BERIDY,
AHMED (AHMED)
Subject: RE: Etisalat UAE/Repeated TMUs reset on RNC231 /1-5532074

Hello Karim

If you will set the "isEnhancedIurRabConfigEnabled" to false, the RNC will

refuse
RABs with faulty SRB. Impact to KPI is related only to RABs, which will be
refused. So here is assumption, that IuR traffic will start to be a little
bit
lower.
Set this feature to false and monitor KPIs, please. In case there will be
unexpected impact to KPI, we will report this matter to TEC.

Best Regards

Robert Ries

16. 06-Mar-2015 21:51

rories

From: RIES, Robert (Robert)

Sent: Friday, March 06, 2015 6:50 PM
To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **
Cc: BERIDY, AHMED (AHMED); BOSLEY, Tim (Tim)** CTR **; KHEDR, MAHMOUD
(MAHMOUD);
REDA, RAMY (RAMY); ABDEL-HALIM, SAYED (SAYED); MOHAMED, GEHAD (GEHAD); ELMIDANY,
AHMED (AHMED); OKASHA, TAHER (TAHER); MERAD, ABDESSAMAD (ABDESSAMAD);
BERIDY,
AHMED (AHMED)
Subject: RE: Etisalat UAE/Repeated TMUs reset on RNC231 /1-5532074

Hello Karim

You asked us for reminder once the fix is implemented. So now is fix for
this
issue implemented in LR13.3, LR14.2, LR14.2.5, LR15.3 loads.
As you already authorized, kindly I will close this ticket.

Best Regards

Robert Ries

17. 12-Mar-2015 11:31

rories

ALCATEL-LUCENT PROPRIETARY

Time Tracking Entry Added - IR01-TSA3

18. 12-Mar-2015 11:33

rories

Update to Current Summary: Fix in LR13.3, LR14.2, LR14.2.5, LR15.3 loads

19. 12-Mar-2015 11:33

rories

ALCATEL-LUCENT PROPRIETARY

Time Tracking Entry Added - IR01-TSA3

20. 12-Mar-2015 11:34

rories

ALCATEL-LUCENT PROPRIETARY

Time Tracking Entry Added - IR01-TSA3

Resolution
Fix for this issue implemented in LR13.3, LR14.2, LR14.2.5, LR15.3 loads
End of Assistance Request 1-5532074
New AR:

View AR | AR without Proprietary | OLCS

History | States | Assignment | Timetracking
AR Text Search | More | CARES Home