CSCtj50884 Bug Details

/Support/BugTool

Tomcat - java.lang.OutOfMemoryError due to large number of RTMT requests. Symptom: Symptoms include: - Slow response on access to UCM web pages. - Failed Extension Mobility operations due to timeout or unreachable server. - Failed IPCC agent logins due to failing AXL requests. - java.lang.OutOfMemoryError in Tomcat logs (catalina.out). Conditions: UCM 6.1.4, 6.1.5, 7.1.5, 8.0.3, 8.5.1. This issue can be seen even if older versions of the RTMT client are used to connect to a server with the fix. To mitigate this you must download and install the newer RTMT client from CUCM after upgrading to a version with the fix. Note: If RTMT clients, "old" clients, are actively attempting to connect when the UCM nodes come online after an upgrade the memory leak WILL return. It is recommended to stop all RTMT clients before the upgrade is executed, Tomcat access logs will show the IP of any connected clients. After the Publisher is upgraded the updated RTMT client may be used to monitor the upgrade of the remaining nodes. Status Fixed (Verified) Severity 2 - severe Last Modified In Last 7 Days Product Cisco Unified Communications Manager (CallManager) Technology

1st Found-In 6.1(5) 7.1 8.0(3) 8.5(1) 8.6(0.96071.5) Fixed-In 8.6(0.98000.8) 6.1(5.13900.4) 8.6(0.98000.24) 8.5(1.12008.3) 8.6(0.99081.4) 6.1(5.13039.1) 7.1(5.33043.1) 8.0(3.22900.2) 8.6(0.99081.6) 8.0(3.22026.1) 7.0(2.24055.1) 8.6(0.99981.2) 8.6(1.10000.1) 8.6(1.10001.1) 8.6(1.98000.43) 8.6(0.95180.9) 9.0(0.95010.1) 8.6(1.10000.43) 8.6(1.95020.80) 9.0(0.96000.1) Component(s)

Workaround: Restart Tomcat from the CLI of the server with the command utils service restart Cisco Tomcat

Additional Info: To determine if a reboot is needed: To avoid unexpected web service outages, monitor the Cisco Tomcat JVM counters with RTMT or the CLI and proactively restart Tomcat if the TOTAL heap size = MAX heap size. Tomcat will continue to run when TOTAL =

MAX while FREE is > 0. To verify a server is experiencing CSCtj50884: 1.) Download the Tomcat Java heap file (*.hprof) automatically generated on the CUCM server to your laptop or desktop file get activelog tomcat/logs/* 2.) Run the following Java command against the downloaded heap: jhat -J-mx2048m < heap-file>.hprof It will take a few minutes and eventually start a HTTP server on port 7000. Note, jhat is provided as part of the Java Development Kit (Standard Edition). You will need to install the Sun/Oracle JDK locally in order to run the jhat command. http://www.oracle.com/technetwork/java/javase/downloads/index.ht ml 3.) Access the server via any web browser: http://:7000 4.) Select the "Execute Object Query Language (OQL) query" option at the very bottom of the page. 5.) Enter the following OQL query and click the Execute button. select count(heap.objects('org.apache.catalina.session.StandardSession' ), "it.isValid.toString()=='false'") Crashes due to CSCtj50884 will have thousands and thousands of invalid sessions.

rtmt
Related Bug Information

CUCM Web very slow after upgrade 8.6.0.96071-5 Symptom:CUCM GUI very slowConditions:Workaround : Tomcat OutOfMemoryError due to large number of RTMT requests Symptom:Slow response on accessing serviceability pages.Conditions:When accessing Serviceability pages.Workaround:Additiona l Notes: Please refer to CSCtj50884
Interpreting This Bug

Additional Info #2: Eclipse Memory Analyzer (MAT) may also be used to confirm this DDTS, http://www.eclipse.org/mat/

Bug Toolkit provides access to the latest raw bug data so you have the earliest possible knowledge of bugs that may affect your network, avoiding un-necessary downtime or inconvenience. Because you are viewing a live database, sometimes the information provided is not yet complete or adequately documented. To help you interpret this bug data, we suggest the following: The status of this bug is Verified. The Fix for the problem is tested. This is the final resting place for all fixed bug reports that are verified by a test engineer. Normally, the test engineer moves a bug report into this state. This status is rarely used. The fix is contained in the next major release following any or all of the versions listed in the "fixed-in"

Find major memory user: 1) Download & install MAT, http://www.eclipse.org/mat 2) Open heap dump in MAT 3) Select "Leak Suspects Report" & "Finish"

Find number of outstanding connections: 1) Open the Open Object Query Language studio 2) Execute this query, SELECT o.id.toString(), o.isValid, o.creationTime FROM org.apache.catalina.session.StandardSession o WHERE (o.isValid=false)

If the major user is "org.apache.catalina.authenticator.SingleSignOn" & the number of outstanding connections (total entries) are in the 1000's or 10's of 1000's CSCtj50884 is root cause of the failure.

section. Any "workaround" listed in the bug details section is generally provided as a way to circumvent the bug until the code fix has been completed; often in lieu of downgrading to a non-affected version of code. Sometimes the bug details, when available, contain the "fixed-in" version information or link to the upgrade or patch. Check for a release later than these listed versions in software download center. The "fixed-in" version may not be available for download yet until all the other bugs targeted to be fixed for that major release are processed. No release date information is available to Bug Toolkit. Please check the software download section frequently to look for a new version. Always check the software release notes before performing any upgrade to understand new functionality and open bugs not yet fixed. In certain rare circumstances, we are unable to fix the bug in all versions in which it is found. The bug will still have a 'fixed' status. Please

open a service request with the Technical Assistance Center if you are being impacted by a bug in this condition. You may select "Save Bug" and setup an alert to be notified about future updates to this bug. This bug has a Severe severity level 2 designation. Important functions are unusable but the router's other functions and the rest of the network is operating normally. Severity levels are designated by the engineering teams working on the bug. Severity is not an indication of customer priority which is another value used by engineering teams to determine overall customer impact. Bug documentation often assumes intermediate to advanced troubleshooting and diagnosis knowledge. Novice users are encouraged to seek fully documented support documents and/or utilize other support options available.