Troubleshooting a Domino Server Crash

Open Mic Webcast August 31, 2011

2011 IBM Corporation

Agenda

W hat is a server crash? Server debug Using IBM Support Assistant (ISA) Gathering files for troubleshooting (MustGather) Lotus Diagnostic Tool Q&A

2011 IBM Corporation

What is a server crash...

A Domino server crash is a situation where one of the server processes has terminated and it is no longer running. A crash could also be any process that the Domino Server has launched, a Lotus companion product or third party product that is accessing the Domino Server partition that crashes in such a way that causes the Domino Server to become inaccessible. Common symptoms of a Domino server crash include: The Domino server is no longer running, but other programs on the system are still running. The Domino server console does not appear, even when tasks appear to be loaded. The Domino server loaded and abruptly came down without doing anything. A panic error appears on the console or in Log.nsf, and the system comes down. NSD/RIP automatically runs and generates a file, and the server comes down and/or restarts by itself. Users cannot access the server

2011 IBM Corporation

Server Debug

Typical Basic Notes.ini debugging parameters: Debug_threadid=1 logs each process and thread ID for each server operation Debug_show_timeout=1 turns on semaphore timeout messages to the console, and creates a semaphore text file called semdebug.txt Debug_capture_timeout=1 time stamps each semaphore timeout message CONSOLE_LOG_ENABLED=1 (Domino 6 and later) enables Domino console logging

In 8.5.3 these are enabled by default

2011 IBM Corporation

ADC and Fault Analyzer

Automatic Data Collection (ADC) Notes/Domino 6.0.1 introduced the automatic diagnostic data collection tool, also known as Automatic Data Collection, or ADC for short. Automatic Data Collection simply means that when a Notes client or Domino server crashes, the program gathers all the necessary data to debug the crash and sends it to a mail-in database when the client or server restarts. Administrators then have one location per domain in which they can see all the crashes that have occurred for all clients and servers. This will help eliminate the instances where an administrator or user may not be able to capture the proper data on a client or server crash. More info: What is the Automatic Diagnostic Data Collection tool?http:// www.ibm.com/support/docview.wss?uid=swg21085850

Fault Analyzer Fault Analyzer can be enabled to correlate common crashes on a server and between servers to let you see which problem is affecting your Domain the most and to help prioritize these with Lotus Support.

Fault recovery can be setup to quickly recover from Domino server crashes.
2011 IBM Corporation 5

Gathering files for troubleshooting (MustGather)

Automated log collection via IBM Support Assistant (ISA) is the recommended method for improving accuracy of the data collection. More info: Collecting Data Read first for Lotus Notes & Dominohttp:// www.ibm.com/support/docview.wss?rs=899&uid=swg21415777 Collect the notes.ini files in the Lotus Domino install directory. Collect the following files in the IBM_TECHNICAL_SUPPORT directory:
latest NSDs (*.nsd or *.log) dated for the day the problem occurred latest console.log (*.log) associated with the NSDs collected the latest process*.dmp (if available) semdebug.txt (if available) logasio*.txt files

2011 IBM Corporation

Lotus Diagnostic Tool

Lotus Notes Diagnostic tool is used to perform repetitive tasks associated with analyzing files from a Lotus Domino server crash, hang, and performance issues. It runs on Microsoft Windows but can analyze Domino Server files from any platform. Lotus Notes Diagnostic is available within the IBM Support Assistant The IBM Support Assistant provides a tooling platform for you to easily download and use various problem determination and analysis tools at no charge. Below you will find a selection of IBM Support Assistant Tool Add-ons. How to Install ISA-Lite http://www.ibm.com/support/docview.wss?uid=swg24025506 How to Intsall add-ons Lotus Diagnostic Tool http://www.ibm.com/support/docview.wss?uid=swg27013116

2011 IBM Corporation

Legal Disclaimer
IBM Corporation 2011. All Rights Reserved. The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBMs current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/ or capabilities referenced in this presentation may change at any time at IBMs sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Sametime and WebSphere are trademarks of International Business Machines Corporation in the United States, other countries, or both.

2011 IBM Corporation