Diagnostics Agent Troubleshooting

SAP Solution Manager 7.0 EHP1, 7.
10
End-to-End Diagnostics
Version 3.0, August 2013
Diagnostics Agent Troubleshooting Guide
END-TO-END DIAGNOSTICS
DIAGNOSTICS AGENT TROUBLESHOOTING
Type Style Example Text
Represents Words or characters quoted from the screen. These include field names, screen titles, pushbuttons labels, menu names, menu paths, and menu options. Cross-references to other documentation.
Icon
Meaning Caution Example Note Recommendation Syntax
Example text
EXAMPLE TEXT
Emphasized words or phrases in body text, graphic titles, and table titles. Technical names of system objects. These include report names, program names, transaction codes, table names, and key concepts of a programming language when they are surrounded by body text, for example, SELECT and INCLUDE. Output on the screen. This includes file and directory names and their paths, messages, names of variables and parameters, source text, and names of installation, upgrade and database tools. Exact user entry. These are words or characters that you enter in the system exactly as they appear in the documentation. Variable user entry. Angle brackets indicate that you replace these words and characters with appropriate entries to make entries in the system. Keys on the keyboard, for example, F2 or ENTER.
Example text
Example text
<Example text>
EXAMPLE TEXT
END-TO-END DIAGNOSTICS
DIAGNOSTICS AGENT TROUBLESHOOTING
Contents
1 Document Status ................................................................................................................................. 5 1.1 New Document Location .............................................................................................................. 5 1.2 Location of Intermediate Document ............................................................................................ 5 1.3 Old Document Location ................................................................................................................ 5 Pre-requisites ...................................................................................................................................... 6 2.1 JDK compatibility .......................................................................................................................... 6 2.2 Managed System ........................................................................................................................... 6 2.3 SLD production system ................................................................................................................ 7 Agent installation strategy ................................................................................................................ 10 3.1 Managing system scaling ........................................................................................................... 10 3.2 Managed System environment ................................................................................................... 10 3.3 Managed Systems HA/Failover environment ............................................................................ 10 Agent installer.................................................................................................................................... 11 4.1 SAP Host Agent........................................................................................................................... 11 4.1.1 E2E RCA configuration for Solution Manager 7.0x or higher .................................................. 11 4.1.2 E2E RCA configuration for Solution Manager 7.1x only.......................................................... 11 4.2 OS configuration checks ............................................................................................................ 12 4.3 JVM parameters .......................................................................................................................... 12 4.4 Startup scripts ............................................................................................................................. 13 Agent connectivity ............................................................................................................................ 17 5.1 Managing system ........................................................................................................................ 17 5.2 Managed system ......................................................................................................................... 19 Connection configuration for managed systems .................................................................................. 22 5.3 Self-Management ........................................................................................................................ 22 5.4 SAP router ................................................................................................................................... 23 Agent Maintenance ............................................................................................................................ 24 6.1 Configuration............................................................................................................................... 24 6.1.1 Change Path of the JDK......................................................................................................... 24 6.1.2 Change JVM Parameters ...................................................................................................... 24 6.1.3 Replacing JDKs 1.4.2 with SAP JVM 4 .................................................................................. 24 6.1.4 Verifying Logical hostname .................................................................................................... 25 6.1.5 Usage of smdsetup script ....................................................................................................... 25 6.1.6 Change Agent log level .......................................................................................................... 28 6.2 Managing J2EE user ................................................................................................................... 29 Tracking Agent issues ...................................................................................................................... 30 7.1 OSS message pre-requisites ...................................................................................................... 30 7.2 Agent crash investigation........................................................................................................... 30 7.3 Thread dump ............................................................................................................................... 32 7.4 Activate P4 logs .......................................................................................................................... 33 Scalability........................................................................................................................................... 34
End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide
8.1 Solution Manager J2EE stack settings ...................................................................................... 34 9 Telnet SMD server commands.......................................................................................................... 35 10 Full Qualified Name resolution issue ............................................................................................... 36
SAP AG 2013
1
1.1
Document Status
New Document Location
Here http://wiki.sdn.sap.com/wiki/display/SMSETUP/Diagnostics+Agent+Troubleshooting
The new location for the troubleshooting content originally covered by this document is on the SDN WIKI
(also accessible via this tiny link http://wiki.sdn.sap.com/wiki/x/aQSAF)
1.2
Location of Intermediate Document
While we are reviewing the content of this document, and migrate it to the SDN WIKI, this intermediate version of this document will be available as an attachment to SAP Note 1833501, as DiagnosticsAgentTroubleshooting.pdf.
1.3
Old Document Location
The old location of this document is on the SMP (both this SPM location and the document on SMP are deprecated/obsolete) https://service.sap.com/~sapdownload/002007974700000409092009E/DiagAgent_TroubleShooting.pdf
SAP AG 2013
2
2.1
Pre-requisites
JDK compatibility
The following table describes the list of different JDKs supported by the Diagnostics Agent for each OS platform:
OS Server 2003/2008 5.3 64bit on Power 4, 5 6.x 64bit on Power 4, 5 11.31 on IA64 64bit 11.11, PA-RISC 64bit 10 on SPARC 64bit / x86_64 IA32 32bit / IA64 64bit x86_64 64bit Power 64bit SLES9 or higer on zSeries 64bit V5R3, V5R4 64bit 1.6, 1.7, 1.8, 1.9 64bit Diagnostics Agent 7.00 / 7.01 Diagnostics Agent 7.11 Diagnostics Agent 7.20 Diagnostics Agent 7.30 SP2
Windows
Note 716604 Note 716927 Note 716927 Note 716926 Note 716926 SAPJVM5 Note 716604 Note 716604 Note 861215 Note 810008 Note 746299 Note 717376 SAPJVM6 SAPJVM6
AIX
HP
Solaris
Linux 2
I5/OS Z/OS
The Diagnostics Agent installation packages 7.1x, 7.20 and 7.30 include the SAPJVM and the SAP Kernel DVD. Therefore, it is not necessary to install separately the SAPJVM before installing one of those Agents. The above listed SAP notes could be helpful to determine the adequate JDK version but do not apply JVM parameters proposed in these SAP notes for the Diagnostics Agent. (They are only intended to SAP J2EE Engines).
2.2
Managed System
The following list describes the SAP systems for which SAPInst installs a Diagnostics Agent at the end of the SAP Instance (central or dialog) installation.
System SAP NW 7.0/BS 2005(ERP/CRM/SCM/SRM 5.0 SR3) based on AS ABAP SAP NW 7.0/BS 2005(ERP/CRM/SCM/SRM 5.0 SR3) based on AS Java/Java + ABAP SAP Solution Manager 7.0 (SR4) SAP NW 7.01 SR1/BS 2008 SR1(ERP 6.00 EHP4) based on AS ABAP SAP NW 7.01 SR1/BS 2008 SR1(ERP 6.00 EHP4) based on AS Java/Java + ABAP SMD SAPinst release SMD SAPinst 7.0 (SR3) SMD SAPinst 7.01 SR1
Read SAP note 171356 before to install the agent
SAP AG 2013

SAP NW 7.1 (SAP PI/CE) SAP NW 7.11 (SAP PI/CE/Mobile/AS ABAP/AS Java/Web Dispatcher/Workplace) SAP NW 7.2 (SAP CE/AS ABAP/AS Java/Web Dispatcher/Workplace) SAP NW 7.3 (AS ABAP/AS Java/Web Dispatcher/Workplace) SMD SAPinst 7.11 SMD SAPinst 7.2/7.30 SMD SAPinst 7.2/7.30
IMPORTANT: In any case refer to the SAP note 1365123 and its attached document.
2.3
SLD production system
The SLD CIM Model version must be 1.5.26 or higher to be able to register a Diagnostics Agent. How to determine if I need to register my agent in SLD or not? Refer to the SAP note 1365123 and its attached document How to determine which version of CIM Model is installed? Go in SLD with url http://sldhostname:port/sld Logon with J2EE user with role SAP_SLD_ADMINISTRATOR Click on link About, the CIM Model version is displayed in a window popup. How to upgrade the CIM Model? Refer to SAP Note 669669 / Updating the SAP Component Repository in the SLD
SAP AG 2013
End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide How to check if the Diagnostics Agent is known in a given SLD in order to connect it to Solution Manager? Go to the SLD with URL http://sldhostname:port/sld Logon with J2EE user with role SAP_SLD_ADMINISTRATOR Go to the Administration section For SLDs based on NW 7.10 and higher, click on the link Maintenance / CIM Instances For SLDs based on earlier NW releases, click on the link Content Maintenance Select the subset: All with Instances and Class: SAP_DiagnosticsAgentInstance N.B.: No Diagnostics Agent is currently registered, in case the class Diagnostics Agent Instance does not appear in drop down list. In column Name of table, check if the desired Diagnostics Agent is present. Or since Solution Manager 7.1 use rather the step Connect Diagnostics Agent in transaction solman_setup, scenario System Preparation This step displays for the selected SLD the list of known Diagnostics Agents Remind that in case the landscape includes several SLDs, having an SLD bridge forwarding in place, the above approach will ONLY allow validating that a specific Diagnostics Agent is registered in one of those SLDs. But it does NOT necessarily mean that this Agent is directly registered in the precise SLD, where the look-up is been performed, like explained above. This means, to be able to connect Diagnostics Agents using the Connect Diagnostics Agents feature in solman_setup transaction / System Preparation or in Managed System Configuration / Assign Diagnostics Agent, it is important to select the SLD in which the Diagnostics Agent is directly registered. This direct registration target SLD is decided and configured in most cases at Diagnostics Agent installation time. N.B.: Remind to have a look at the runtime.properties file of the Diagnostics Agent, in order to find out in which SLD the Agent is directly registered in.
Which J2EE Group is assigned to the SLD user used by the Diagnostics Agent? The J2EE SAP_SLD_DATA_SUPPLIER group or the security role DataSupplierLD should be assigned to the user used by the Diagnostics Agent. What is the SLD registration frequency of Diagnostics Agent? Every 12 hours, the Diagnostics agent sends an SLD request to update the agent model information. What is the frequency on which the Diagnostics Agent checks within its associated SLD for Solution Manager connectivity data updates? If the Diagnostics agent is registered in an SLD and not yet connected to a Solution Manager System, it will check every 20 minutes if an association has been created in that SLD, using the Connect Diagnostics Agents feature in solman_setup transaction / System Preparation or in Managed System Configuration / Assign Diagnostics Agent. If the Diagnostics agent registered in an SLD and currently connected to a Solution Manager System, it is going to check every hour if a new / modified association has been created in that SLD.
SAP AG 2013
End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide Why is the Diagnostics Agent not visible in the Agent Administration UI? Verify the below listed items, in case the Diagnostics Agent is not connecting to the Solution Manager system after you used the Connect Diagnostics Agents feature, via solman_setup transaction / scenario: Managed System Configuration / Step: Assign Diagnostics Agent, or scenario: System Preparation. N.B.: Make sure to also search the Agent entry in the Agent Administration UI, using the filter function on the Host name column (visible when the More details button is enabled). Also, neither use the Server name (as it might be not assigned yet), nor the Fully Qualified Hostname (as hostname resolution issues will for example make the UI display the physical host) as filter criteria. You can access the Agent Administration UI in general via a hyperlink displayed in the above mentioned solman_setup steps. o o o o Has the adequate SLD been selected when performing the Connect Diagnostics Agents operation? See also above Q&A item. Did you wait long enough? See also above Q&A item. Is the Diagnostics Agent still able to interact with its associated SLD? Check for SLD CIM or SLD credentials errors in the Diagnostics Agent SMDSystem.?.log log files Is the Diagnostics Agent detecting a Solution Manager connectivity data change in the SLD? Check for other kind of issues in the Diagnostics Agent SMDSystem.?.log log files. Alternatively simply connect the Diagnostics Agent to the Solution Manager system with the script smdsetup (see part 6.1.5 Usage of smdsetup script) instead of using indirectly the SLD to do so.
Note that in case the following message appears in former Agent Candidates UI : 2010.03.17 14:50:26, Managing System '<SID>.SystemHome.<host>' is registered in the SLD but
the CIM declaration is not valid, the agents cannot be attached correctly. Check SLD Managing system declaration. (detail: CIM_ERR_FAILED: No such instance: sld/active:SAP_J2EEEngineCluster.CreationClassName="SAP_J2EEEngineCluster",Name="<SID>.Syst emHome.<host>")
This error simply reports that Solution Manager instance is not found in SLD with the mentioned hostname BUT it does not prevent the agent attachment process to succeed. The UI will ask for the hostname and port of Solution Manager during the attachment sequence. This error message is solved in 7.01 SP7 P1 or higher.
SAP AG 2013
10
Agent installation strategy

3.1 Managing system scaling
Use the SAP Solution Manager Sizing Toolkit to check if is necessary to configure your solution manager system as cluster environment. If it is necessary we strongly recommend using additional Dialog instances instead of adding Java server nodes to already existing Instances (like 1 dispatcher / 2 or more server node). Because the dispatcher is the critical component to connect the Agents to Solution Manager, it is more interesting to have many dispatchers (and therefore instances) for failover/load balancing capabilities. Last but not least also consider the currently known limitation documented in SAP Note 1480419 / Limitations in Alerting and Monitoring
3.2
Managed System environment
To determine the Agent version and the amount of Agents to install (on Managed System hosts) refer to SAP Note 1365123 The following check list must be validated before installing a new Diagnostics agent on a managed host: Check whether a Diagnostics Agent is already installed: Verify if the managed system has been installed with a SAPInst version greater than NW04S SR3 (see part 2.2 Managed System) or check on the file system whether a folder like /usr/sap/DAA or /usr/sap/SMD already exists. In both cases an agent is already installed on the system. If the Diagnostics Agent(s) is/are already installed on the Managed system host(s), check if the Agent(s) is/are present in Agent Administration UI. If not, double check whether the Agent(s) is/are known in the SLD where the managed system is registered. In this last case connect the Agent to the Solution Manager System using the Connect Diagnostics Agents feature (or the Agent candidate Management with Solution Manager 7.0 EHP1). Alternatively simply connect the Diagnostics Agent to the Solution Manager system with the script smdsetup (see part 6.1.5 Usage of smdsetup script) instead of using indirectly the SLD to do so.
In case no Diagnostics Agent is installed, first find out the logical hostname or physical/virtual hostname of the managed system. Install the Diagnostics Agent following the rules defined in the document of SAP Note 1365123, on
3.3
Managed Systems HA/Failover environment
With Solution Manager 7.1 SP05 and higher refer to the following article to put in place the so called Agents On-the-fly: Diagnostics+Agent+and+HA+support. In the context of earlier Solution Manager Releases, it is recommended to install the Diagnostics Agent on shared/cluster file system. The goal is to move the Agent each time together with the Managed System (only for Application Server Instances, not ASCS/SCS), but adding the Agent stop, file system un-mounting, file system remounting and re-start in the cluster fail-over scripts. In case the Managed instance and Agent are installed on different file system partitions, it is important when the Managed SAP instance and the Diagnostics Agent are moved to another cluster node to mount the Agent Instance file partition before the SAP Instance partition, to allow the Introscope Byte Code Adapter to start . This is due to the fact that the Wily Setup wizard makes the SAP java instance classpath depending on some jar files contained in the Diagnostics agent folder, especially if the Introscope Byte Code Adapter has been setup with Solution Manager (prior to Solution Manager 7.1 SP05). See also Diagnostics+Agent+and+HA+support for further details.
SAP AG 2013
11
4
4.1
Agent installer
SAP Host Agent
Startup commands
The SAPHOSTAGENT is located in /usr/sap/hostctrl/exe (on UNIX) On Windows it is usually located in C:\Progam Files\SAP\hostctrl\exe In general you need to work with the root or Administrator user to execute the SAPHOSTAGENT commands Action restart SAPHOSTAGENT Check its status Check its version Command saphostexec -restart saphostexec -status saphostexec version
On Windows it is usually installed as a service, meaning that it can be started or stopped via Windows Services
4.1.1
E2E RCA configuration for Solution Manager 7.0x or higher
To allow the Diagnostics Agent to retrieve CPU, Memory and disk usage via the SAP Host Agent, it is necessary to check the SAP Host Agent profile is configured as follow:
Location : "C:\Program Files\SAP\hostctrl\exe\host_profile" or host_profile /usr/sap/hostctrl/exe/
Property to enable the different webservice port type: service/porttypes = SAPHostControl SAPOscol SAPCCMS SAPControl
4.1.2
E2E RCA configuration for Solution Manager 7.1x only
For some E2E RCA applications like LogViewer it is necessary to update the SAP Host Agent profile to add the OS user of Diagnostics Agent as user authorized to call specific protected methods via dedicated authentication. Therefore add or update the property service/admin_users in the SAP Host Agent profile "C:\Program Files\SAP\hostctrl\exe\host_profile" or /usr/sap/hostctrl/exe/ host_profile to list all Diagnostics Agent OS users that are running on that Physical or Virtual host:
For Windows: service/admin_users= <DOMAIN\USER> <DOMAIN\USER>
ex:
service/admin_users= MY_DOMAIN\SAPServiceDAA MY_DOMAIN\SAPServiceDAB For UNIX: service/admin_users=
<SIDADM USER> <SIDADM USER>
ex:
service/admin_users= daaadm dabadm
SAP AG 2013
12
4.2
OS configuration checks
After the installation of the Diagnostics Agent under UNIX platform, check the following OS parameters of the Diagnostics Agent OS user (ex: daaadm) Check the Diagnostics agent OS user limit : Logon with the agent OS user su smdadm Open an sh shell with the command sh Execute the command ulimit -a Compare the values with the following recommended values Limit time(seconds) file(blocks) data(kbytes) stack(kbytes) memory(Kbytes) coredump(blocks) nofiles(descriptors) Recommended value unlimited unlimited unlimited 4194304 unlimited 4108901 8092
If the users limit parameters for the OS Diagnostics Agent user are not correct, some OutOfMemory exceptions can for example occur during the extraction activities performed by the Diagnostics Agent. Also check the note 1437105. Once the limits are modified also remember to restart the Diagnostics Agent system processes (sapstartsrv) so that the changes are taken into account. For AIX system see note 323816. If during a zip extractor job (datacollect), an access denied error occurs (could be found in the Diagnostics Agent log SMDApplication.?.log), double check if the folder /usr/sap/<SID_AGT>/JXX/SMDAgent/temp is writable for the sapsys group. Furthermore the file system of the Managed system must be readable by the Diagnostics agent. Logon with an agent OS user (su smdadm), go in the managed system folder (ex: /usr/sap/SID/SYS/profile), and open one profile with the command more <SID>_<INSTANCE_ID>_<HOST>. If the more command is not working, the access rights are not correct for the Diagnostics agents. In that case, check with your system Administrator to adapt the file system rights. Further details could be found in SAP note 1163751.
4.3
JVM parameters
In this section, you will find the recommended JVM parameters for each JDK vendor 1.4 of older Diagnostics Agents 7.00/7.01: JDK JVM Parameters SAP Note -DP4ClassLoad=P4Connection -Xj9 -Xgcpolicy:gencon -Xmn50m -Xmx256m 1093831 IBM JDK -Xms256m SUN JDK 32 bits -DP4ClassLoad=P4Connection -Xmx128m -Xms128m SUN JDK 64 bits -DP4ClassLoad=P4Connection Xmx256m Xms256m n.a HP JDK 32 bits -DP4ClassLoad=P4Connection -Xmx128m -Xms128m HP JDK 64 bits -DP4ClassLoad=P4Connection Xmx256m Xms256m
SAP AG 2013
13
With agent 7.1x or higher an SAP JVM is used to execute the Diagnostics Agent, the recommended JVM parameters are: JDK JVM Parameters SAP JVM4/5/6 32 bits -DP4ClassLoad=P4Connection -Xmx128m -Xms128m -XX:MaxPermSize=64m SAP JVM4/5/6 64 bits -DP4ClassLoad=P4Connection Xmx256m Xms256m -XX:MaxPermSize=128m Dont put the java parameter verbose:gc in the Diagnostics Agents JVM parameters, because this parameter will steadily increase the size of the file work/std_SMDAgent.out
4.4
Startup scripts
Information about Diagnostics Agents 7.11 or higher

With Diagnostics Agents 7.11, 7.20 & 7.30 the startup system is compliant to the SAP startup mechanism. It means that the former commands smdstart.sh and smdstop.sh are no longer available. Under Windows the SAP MMC allows to start or stop the Diagnostics Agent. The Windows service of the Diagnostics Agent is configured to start automatically during the Windows startup.
The green status in SAP MMC only indicates that the Diagnostics Agent process is running. It does however NOT mean that the Agent is operational / connected to the Solution Manager system. Only the Solution Manager Agent Administration UI allows to confirm that the Diagnostics Agent is successfully connected to the Solution Manager system and operational. (In the status is gray see 7.2 Agent crash investigation)
Under UNIX, while being logged with the relevant Diagnostics Agent OS User, use the following commands to operate the Diagnostics Agent. Command startsap SMDA<Instance_Number> [-v <logical hostname>] stopsap SMDA<Instance_Number> [-v <logical hostname>] Action Start the Diagnostics agent. Stop the Diagnostics agent.
The sapstartsrv process of Diagnostics Agent is automatically started during the boot sequence of system if the sapinit program has been installed on the system. The Diagnostics Agents could later be also stopped and started remotely with the help of the SAP MC or SAP MMC.
Information about former Diagnostics Agents 7.0x

On Windows platforms the SAP MMC allows to start or stop the Diagnostics Agent. The Windows service of the Diagnostics Agent is configured to start automatically during Windows startup.
The green status in SAP MMC only indicates that the Diagnostics Agent process is running. It does however NOT mean that the Agent is operational / connected to the Solution Manager system. Only the Solution Manager Agent Administration UI allows to confirm that the Diagnostics Agent is successfully connected to the Solution Manager system and operational. (In the status is gray see 7.2 Agent crash investigation) On UNIX platform, the commands smdstart.sh, smdstop.sh and smdadmin.sh allow to operate the Diagnostics Agent. Since 7.0 release of the Diagnostics Agent SAPInst, those commands are included in the PATH
SAP AG 2013
End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide variable of the OS agent user. In that case, the commands can be called from every folder (not only in the Agent script subfolder). Command smdstart.sh <SID> <InstanceNumber> smdstop.sh <SID> <InstanceNumber> smdadmin.sh <SID> <instanceNumber> Action Starts the Diagnostics agent and the sapstartsrv process. Stops the Diagnostics agent and the sapstartsrv process Launches jcmon tool to administrate the Diagnostics agent instance
14
On Solaris platform, an error (/usr/sap/SMD/exe/smdstart.sh: usage: ulimit [ -HSacdfnstv ] [ limit ]) occurs when the smdstart.sh is executed. Check the solution in sap note 1109366. On HP-UNIX platform, an error (!! ERRROR !! current user is not the owner of this instance ) occurs when the smdstart.sh is executed. Check the solution in sap note 1109366.
Exit code of smdstart.sh 0 2 5 4 3
Reason Success to launch the agent process (This is not the confirmation that the agent is connected to SMD). Input Parameters are invalid. Agent is already running. User limit are not set correctly. Current OS user is invalided to start the agent.
SAP AG 2013
15
For UNIX system, the Diagnostics Agents do not start automatically when the system reboots because the Diagnostics Agent startup scripts are not included in the system startup scripts. The following command line can be used in the system startup scripts: su -c /usr/sap/SMD/J98/script/smdstart.sh > /dev/null 2>&1 And in the shutdown sequence: -s /bin/sh - smdadm
su -c /usr/sap/SMD/J98/script/smdstop.sh > /dev/null 2>&1
-s /bin/sh - smdadm
The following procedure describes how to insert the agent in the boot sequence for a Linux SUSE 10 system: Create a file diagnostics_agents in folder /etc/init.d Determine the init run level in file /etc/inittab Create a symbolic link in folder /etc/init.d/rc<INIT_LEVEL>.d/ to file diagnostics_agents with name S80diagnostics_agents Create a symbolic link in folder /etc/init.d/rc6.d to file diagnostics_agents with name S80diagnostics_agents. (runlevel 6 is System reboot) #!/bin/sh ### BEGIN INIT INFO # Provides: diagnostics_agents # Required-Start: $network # Required-Stop: $network # Default-Start: 2 3 5 # Description: start diagnostics agents ### END INIT INFO case "$1" in 'start') su -c /usr/sap/SMD/J98/script/smdstart.sh > /dev/null 2>&1 -s /bin/sh - smdadm ;; 'stop') su -c /usr/sap/SMD/J98/script/smdstop.sh > /dev/null -s /bin/sh - smdadm ;; 'status') su -c /usr/sap/SMD/J98/script/smdadmin.sh -s /bin/sh - smdadm ;; *) echo "Usage: $0 { start | stop | status }" ;; esac exit 0
When smdstart.sh and smdstop.sh commands are embedded in the system startup scripts, the exit code of the smd scripts cannot be retrieved. Further more it is important to redirect the standard error output into the standard output with a syntax like 2>&1 to avoid that the smdstart.sh and smdstop.sh are blocking the parent script.
SAP AG 2013
16
About sapstartsrv process

SAP Note 927637 - Web service authentication in sapstartsrv as of Release 7.00 Note 823941 - SAP Start Service on Unix platforms Description The Web service interface of sapstartsrv requires a user authentication from the client (MMC, SAP MC, sapcontrol and so on) for protected operations such as starting and stopping the SAP instance. After an SAP upgrade, the SAP Start Service (sapstartsrv) on UNIX platforms is not automatically started when the operating system is started.
SAP AG 2013
17
Agent connectivity
5.1 Managing system
This section describes the different primary connection modes used by the Diagnostics Agents to connect to the Solution Manager System. Two modes are available: Connection through the Solution Manager SAPJ2EE Message server Direct connection with the Solution Manager SAPJ2EE P4 dispatcher. Connection through the Solution Manager SAPJ2EE Message server It is RECOMMENDED to connect the Diagnostics Agent to Solution Manager Java stacks using the J2EE Message Server. To establish this connection the hostname and port of the Solution Manager J2EE Message server (Java SCS instance) have to be found out using for example the SCS instance profile.
The Agent connection mechanism can be described mainly as follow: A. During the installation of a Diagnostic Agent the SAPJ2EE Message server connection mode has to be selected. B. This information is stored in the Agent configuration (file runtime.properties) and the credentials are stored in the Agent secure store (file secstore.properties). C. When the Diagnostics Agent starts, the P4 connection framework will retrieve from the SCS Message Server (on HTT port ex: 81xx) all J2EE entry points that are available. D. Once the list of J2EE entry points is collected, one P4 entry point is chosen from the list. E. The Diagnostics Agent will then establish a P4 connection to the selected J2EE entry point with the host name and the port corresponding to that P4 connection.
SAP AG 2013
18
Troubleshooting Message server connection Before starting the Agent installation or before executing the smdsetup script, you can double check the content of the Java SCS Message Server with the following description: 1. Open the URL: http://msgserver:81xx/msgserver/text/logon (xx corresponding to instance number of the SCS instance) 2. Check if a P4 entry point is available with the adequate hostname 3. On the managed system host, open a shell as <SID>adm OS user (from Managed system or Agent) 4. Execute the command ping <host of P4 entry point>. Double check that the host is accessible 5. Execute also the command telnet <host of P4 entry point> <P4 port>. If no error message occurs (blank screen) the connection is OK. Most common agent connection problems All connection problems are logged in a file under SMDAgent/log/SMDSystem.X.log. Check this log file to find the root exception of the issue: Once the error exception is found also locate the root cause of the exception by looking at the last caused by sentence. No Entry point found
Exception: com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exception while trying to get InitialContext. [Root exception is com.sap.engine.interfaces.cross.DestinationException: there are no access points for service: P4 registered on the message server]
No P4 entry point has been reported by the Message Server. This is probably due to the fact that it is not the appropriate Message Server. It has to be the Solution Manager Java stack and not the ABAP stack Message Server. Also check the text display of the Message server content using the URL http://msgserver:81xx/msgserver/text/logon. Verify if a P4 entry point is available (as shown below) Unknown host
Exception: [sapms:// yupxxxxx.domain,com:8136] Checking server availability... Connecting to SMD server sapms://yupxxxxx.domain,com:8136 failed - error counter: 0 - com.sap.engine.services.jndi.persistent.exceptions. NamingException: Exception while trying to get InitialContext. [Root exception is com.sap.engine.interfaces.cross.DestinationException: cannot establish connection with any of the available instances #Nested exceptions are: #java.net.UnknownHostException: yupxxxxx: yupxxxxx ]
It is a network resolution problem due to the fact that the selected P4 hostname retrieved from the SCS Message server list cannot be resolved from the Managed system host. Retrieve the list of the P4 entries in the Message server which looks like: version 1.0 J2EE933894500 J2EE NCED6018XXXXA.dhcp.ncel.sap.corp J2EES NCED6018XXXXA.dhcp.ncel.sap.corp P4 NCED6018XXXXA.dhcp.ncel.sap.corp P4S NCED6018XXXXA.dhcp.ncel.sap.corp P4HTTP NCED6018XXXXA.dhcp.ncel.sap.corp
59300 59301 59304 59306 59305
LB=1 LB=1 LB=1 LB=1 LB=1
Execute the command ping <hostname_of_P4_entry_point_5xx04> in a Managed system shell. If the Solution Manager host cannot be reached, then change the host definition in your DNS or in the /etc/hosts. You should ideally fix you engine installation to report the right host in the Message Server.
SAP AG 2013
19
Agent Maintenance Mode (required during LM-SERVICE deployment) As documented in various notes and guides, the Maintenance mode need to be enabled before deploying a new LM-SERVICE Release, Support Package or Patch, as described also in SAP note 1483508. If this operation is not performed the Diagnostics Agent runtime behavior will in most cases be affected. As a consequence the Diagnostics Agent logs might for example contain errors like: [SMDJNDILookupAgentSideImplementation] lookup /smdserverctx/ObjAutoRegistered/SMD_DataEnrichment_Registrator failed [] Object not found in lookup of SMD_DataEnrichement_Registrator
Solution Manager SAPJ2EE P4 dispatcher (SAP does NOT recommend to use this mode)
Solution Manager in cluster environment If the Solution Manager system is installed as a cluster (having Dialog Instances), the direct P4 Agent connection should not be used as the Diagnostics Agent would else always establish a connection to the same Solution Manager Java instance corresponding to the selected P4 host/port. Also from a somehow High Availability point of view, you have to use a connection through Solution Manager J2EE Message server. In that situation, the connection of the Diagnostics Agent will be each time loadbalanced to the different available Java instances.
5.2
Managed system
All connections from the Diagnostics Agent to the Managed systems are executed locally on the same host. It is the reason why it is really imperative to install the Diagnostics Agent on the same Physical or Virtual host than the SAP Managed system. The Agent connects to different SAP System layers as follow: Java stack on port P4 (5xx04) Sapstartsrv http port of Managed System (5xx13) J2EE Message server (39xx and 81xx) Local RFC call to Managed ABAP stack Webservice call to SAP host agent on port 1128 for agent 7.1x
SAP AG 2013
20
Troubleshooting P4 connection problems with Managed system 1. Check on which host the P4 socket is bound on the Managed system a. Run the OS command netstat an to find the entry line containing the P4 port of the managed system:
TCP 0.0.0.0:59304 0.0.0.0:0 LISTENING
Or
tcp 0 0 *.50304 *.* LISTEN
In this example we see that the P4 port is bound for all hosts where a p4 request can be emitted (* or 0.0.0.0), if the port is bound like 10.45.192.204:50304 only the p4 requests emitted from the host 10.45.192.204 will be accepted. It explains why the Diagnostic Agent must be installed on Logical hostname 10.45.192.204 with SAPInst. The situation can be fixed by changing the profile property SAPLOCALHOST (see 6.1.4 Verifying Logical hostname). b. After verifying the port in the OS configuration, check in the J2EE dispatcher configuration that the property bindHost is set to 0.0.0.0. This should be visible with the J2EE configuration tool of the managed system->Instance_xxx->dispatcher->services->p4. If the value is not configured to 0.0.0.0 (which accepts all hosts), add the Agent IP or install the Agent using on one of the listed IP corresponding to logical hostnames running on the SAME underlying physical or virtual host. 2. Check if the Agent can access the J2EE Message Server (SCS) of the Managed system with the host and the http port stored in the Agent configuration (see the next section). For example : Open a shell with the agent OS user (<SID>adm) on the Managed System Execute the command msprot -l -mshost <host_j2ee_msg_srv> -msserv 39<SCS> or 36<SCS> -d J2EE s, the values of host and port must be retrieved from the Agent configuration (see the next section) The command must return the following content :
SAP Message Server Protocol Program, Version 1.3 (built: Jan 24 2008 01:27:19) INFO CLUSTER-ID HOST GROUP-ID STATE J2EE-SERVICE-MASK ----------------------------------------------------------------------------------------------
SAP AG 2013

---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST ---> LIST SERVICE:00000 NAME:ConfigurationManager SERVICE:00001 NAME:_CacheManager SERVICE:00003 NAME:{service_manager_deploy_distributor} SERVICE:00004 NAME:{service_manager_internal_connection} SERVICE:00005 NAME:shell SERVICE:00006 NAME:jmx_notification SERVICE:00007 NAME:p4 SERVICE:00008 NAME:webservices SERVICE:00010 NAME:iiop SERVICE:00011 NAME:r3startup SERVICE:00012 NAME:telnet SERVICE:00013 NAME:log_configurator SERVICE:00014 NAME:jmx SERVICE:00016 NAME:http SERVICE:00022 NAME:com.sap.security.core.ume.service SERVICE:00023 NAME:security SERVICE:00024 NAME:tc~sec~certrevoc~service SERVICE:00025 NAME:servlet_jsp SERVICE:00026 NAME:webdynpro 0000041052 RUNNING 0000041052 RUNNING
21
LIST 0004105250 ncevhost.mydomain LIST 0004105200 ncevhost.mydomain
XXXXXXXXOXXOXXXXXOXXXXXXXXXXXX XXXXXXOXXXXXXXXXXOOOOOOOOOOOOO
In this content, the P4 service is listed as well as the J2EE nodes with hostnames (e.g. ncevhost.mydomain). Check if those hostnames can be resolved on your network by executing the command: ping ncevhost.mydomain 3. Open the J2EE Visual admin tool of the Managed system and connect with the J2EE user used by the Diagnostics Agent (see in the next section the Agent configuration) to check if the user can be authenticated and if it has the Administrator role. 4. Check also the Managed system defaulttrace file to find some exception related to P4 connections.
SAP AG 2013
22
Connection configuration for managed systems

The following table lists all properties generated and updated by the Managed Setup Wizard or solman_setup with Solution Manager 7.0 EHP1 and higher. These properties allow the Diagnostics Agent to connect to the Managed system in order to complete the E2E RCA operations. They are accessible using the Agent Administration UI / Application configuration. Those properties are stored by server name (scope) of the Diagnostics Agent. Connection Type to Port Configuration stored in agelet Managed system
P4 port http port J2EE Message server Intern Port J2EE Message server Http port J2EE Message server host 5xx04 5xx00 39xx 81xx <Hostname> com.sap.smd.agent.application.global.configuration com.sap.smd.agent.application.global.configuration com.sap.smd.agent.application.global.configuration com.sap.smd.agent.application.global.configuration com.sap.smd.agent.application.global.configuration
Property
[SID]/[INB]/sapj2ee/P4/port [SID]/[INB]/sapj2ee/http/port [SID]/sapj2ee/msgserver/port [SID]/[INB]/sapj2ee/http/port [SID]/sapj2ee/msgserver/host
For the credentials : Connection Type to Managed system

J2EE credentials
Configuration stored in agelet

com.sap.smd.agent.application.global.configuration
Property user
[SID]/sapj2ee/com/user
Property password
[SID]/sapj2ee/com/pwd
Connection Type to Managed system

ABAP credentials 7.0x ABAP credentials 7.1x
Configuration stored in agelet
Property client
[SID]/its/client [SID]/abap/client
Property user
Property password
com.sap.smd.agent.application.global .configuration com.sap.smd.agent.application.global .configuration
[SID]/abap/com/user [SID]/abap/com/user
[SID]/abap/com/pwd [SID]/abap/com/pwd
It is recommended to change those values in the Managed system setup wizard or solman_setup with Solution Manager 7.0 EHP1 and higher instead of directly editing them in the Agent Admin tool. Those values could however be changed using the Agent Administration tool in case of troubleshooting/debugging.
5.3
Self-Management
During a Self-management setup of Solution Manager, some special configurations in the different setup steps are needed due to some restrictions about the local Diagnostics Agent connection sharing. Possible error in Managed system Configuration End of stream is reached unexpectedly during input from Socket
Exception: com.sap.smdagent.plugins.connectors.p4.exceptions.P4AuthorizationException: Access is denied: Please make sure the <user> and <password> parameters are correct.; nested exception is: com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exception during getInitialContext operation.No server is running. [Root exception is com.sap.engine.services.security.exceptions.BaseLoginException: Exception in creating new RemoteLoginContext instance.] at com.sap.smd.agent.plugin.connectors.p4.P4ConnectionService.getP4Connection(P4ConnectionServ ice.java:69) ... at java.lang.Thread.run(Thread.java:838) Caused by: com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exception during getInitialContext operation.No server is running. Root exception is
SAP AG 2013
23
com.sap.engine.services.security.exceptions.BaseLoginException: Exception in creating new RemoteLoginContext instance. at com.sap.engine.services.security.remote.login.RemoteLoginContextExt.<init>(RemoteLoginConte xtExt.java:34) ... at java.lang.Thread.run(Thread.java:838) Caused by: java.io.EOFException: End of stream is reached unexpectedly during input from Socket [addr=/56.zzz.xx.xx,port=50204,localport=22326] at com.sap.engine.services.rmi_p4.Connection.run(Connection.java:423)
It is a problem with a Java server level. Two possibilities to fix the issue: (See also SAP Note 1473530) 1. Apply the fix (see SAP note ) on java server and use at least the LM-SERVICE 7.0 EHP1 SP7 P1 2. Use the workaround describes in SAP note (adapt the property bindHost). In a Self-Management scenario there exists another solution Since Solution Manager 7.0 EHP1 SP21, the Diagnostics Agent configuration property enabling the selfmanagement scenario is added automatically by transaction solman_setup / Managed System Configuration to avoid such a problem. It is named <SID>/selfmonitoring/enabled (value: true) and visible in the Agent Administration under comglobal.configuration.
5.4
SAP router
The Diagnostics Agent can be connected to the Managing system via a SAP Router. The 7.30 standalone installers and SWPM based installers support configuring the Diagnostics Agent connection via a SAP Router. Alternatively the SAP route can be maintained after the Diagnostics Agent installation, using the smdsetup script. (See chapter 6.1.5 Usage of smdsetup script). This feature is supported in the context of Solution Manager 7.10 and higher. Also, the automatic activities Introscope Host Adapter and Introscope Byte Code Adapter installation in solman_setup transaction / scenario: Managed system Configuration will use this route provided at the Diagnostics Agent installation/maintenance level. In the context of Solution Manager 7.0 EHP1 this type of connectivity is not supported. Therefore please contact the Diagnostics Agent support team to analyze the infrastructure and evaluate the setup possibilities.
SAP AG 2013
24
Agent Maintenance
6.1 Configuration
This section describes all procedures to update Diagnostics Agent configurations that could not be changed/updated centrally using the Solution Manager system.
6.1.1 Change Path of the JDK

In the eventuality that the JDK installation path changes after the Diagnostics Agent has been installed, you might need to update this JDK path information within the Agent profile file, as follow.
For 7.00 and 7.01

Open the profile <SID_AGT>_J<INSTANCE_NUMBER>_<hostname> in folder /usr/sap/<SID_AGT>/sys/profile. Update the parameter jstartup/vm/home by putting the absolute path to the JDK home. Remove the file /usr/sap/<SID_AGT>/JXX/profile/smd.properties.vmprop. Restart the Diagnostics agent.
For 7.1x , 7.2x and 7.3x

As the Diagnostics Agent 7.1x and higher run using their own SAPJVM, it is not recommended to change the path of the JDK.
6.1.2 Change JVM Parameters

If the JVM parameters need to be changed, you have to update the Diagnostics Agent instance properties file.
For 7.00 and 7.01

Open the file smd.properties in folder /usr/sap/<SID_AGT>/J<Instance_Number>/profile. Update the parameter smdagent.javaParameters (see 4.3 JVM parameters). Remove the file /usr/sap/<SID_AGT>/JXX/profile/smd.properties.vmprop. Restart the Diagnostics Agent.
For 7.1x, 7.2x and 7.3x

Open the file smdagent.properties in the folder /usr/sap/<SID_AGT>/SMDA<Instance_Number>/SMDAgent. Update the parameter smdagent.javaParameters (see 4.3 JVM parameters). Remove the file /usr/sap/<SID_AGT>/JXX/profile/smd.properties.vmprop. Restart the Diagnostics Agent.
6.1.3 Replacing JDKs 1.4.2 with SAP JVM 4

Some Partner JDK 1.4.2 implementations are going out of maintenance. In this context SAP proposes to replace those JDKs with the SAP JVM 4. The manual procedure to be followed concerning the Diagnostics Agents is described in the
SAP AG 2013
End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide SAP JVM Switch Tool guide available under https://service.sap.com/instguides. Especially, consider the chapter 5. Performing the Switch to SAP JVM 4 for the Diagnostics Agent.
25
6.1.4 Verifying Logical hostname

Verify the logical hostname that was used during the Diagnostics Agent installation, in case a Diagnostics Agent is not reporting the expected logical hostname within the Agent Administration UI.
For 7.01, 7.1x, 7.2x and 7.3x

Open the profile <SID_AGT>_J<INSTANCE_NUMBER>_<hostname> in the folder /usr/sap/<SID_AGT>/sys/profile. The parameter SAPLOCALHOST has to be present and contain a short hostname (not the FQN)
6.1.5 Usage of smdsetup script

The script smdsetup allows to update some configuration parameters for Diagnostics Agent. Some options of the script only exist since Solution Manager 7.0 EHP1. Note that this script is automatically patched at the time the Diagnostics Agent connects to the Solution Manager server. This feature has been implemented since Solution Manager 7.0 SP15. To determine the options supported by the smdsetup script, just execute the command smdsetup.sh or smdsetup.bat, without any parameters If the output proposes some action like managingconf (in case the smdsetup version is 7.01 or higher). Example of smdsetup 7.01 or higher output:
>> Help Information ---------------------Script which allows to update some diagnostics agent configuration. smdsetup <action> <parameters> List of actions: * smdsetup secstorej2ee * smdsetup secstoresld * smdsetup sldconf user:"<value>" pwd:"<value>" user:"<value>" pwd:"<value>" hostname:"<fqn>" port:"<http port>" [optional user:"<value>" pwd:"<value>"] hostname:"sapms://<fqn>" port:"<MS httpPort>" [optional user:"<...>" pwd:"<...>"
* smdsetup managingconf servername:"<...>"] * smdsetup managingconf servername:"<value>"]
hostname:"<fqn>" port:"<p4 port>" [optional user:"<value>" pwd:"<value>"
Script Location
SAP AG 2013
End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide In 7.0 SR3 and 7.0x the smdsetup script in this version is located in: /usr/sap/<SID_AGT>/J<Instance_Number>/script In 7.1x and higher the smdsetup script is located under: /usr/sap/<SID_AGT>/SMDA<Instance_Number>/script Action managingconf Sldconf secstorej2ee secstoresld changeservername addsaprouter saprouterpass removesaprouter supportlogs Execute some Agent checks and generate a zip with all Agents logs (including a Agent check report) 7.0 SP17 patch 3 or higher Full Implementation in 7.1, see chapter 5.4 7.01 or higher Description Changes the Solution Manager System (also known as managing system) used by the Diagnostics Agent to connect to. Specify the host and port. Changes the SLD server where the Agent will perform its registration. Provide the SLD host and port. Changes only the J2EE user used to connect to the Solution Manager system. (restart after the changes) Changes only the J2EE user used to register the Agent in the configured SLD system. (restart after the changes) Change the server name of agent. (restart after the changes) Availability
26
7.0 SP14 or higher
MANAGINGCONF SYNTAX
To execute this command you need to stop the agent. The command updates the connection settings to the Solution Manager system: To connect the agent via J2EE Message server http port: N.B.: - Do not confuse the message server HTTP port with the message server internal port which is like 36xx) - Do not forget to prefix the hostname with sapms://
smdsetup managingconf hostname:sapms://myhost.domain.corp port:8101
(In rare circumstances) To connect the agent with a P4 direct connection:

smdsetup managingconf hostname:myhost port:53004
Optionally, you can add the j2ee user information in the command line to update also the agent secure store:
smdsetup managingconf hostname: sapms://myhost.domain,corp port:8101 user:SMD_ADMIN pwd:init1234
SLDCONF SYNTAX
To execute this command you need to stop the agent. The command updates the connection settings to the SLD system of the agent registration: Register/connect the Agent to the SLD system (http port of SLD java stack).
smdsetup sldconf hostname:mySLDhost.domain.corp port:50000
Optionally, you can add the SLD user information in the command line to update the agent secure store:
smdsetup sldconf hostname:mySLDhost.domain.corp port:50000 user:slddsuser pwd:xxxxx
SECSTOREJEE SYNTAX
To execute this command you need to stop the agent. The command updates the credentials of the connection to the Solution Manager system:
SAP AG 2013
27
Update the agent secstore file:

smdsetup secstorej2ee user:SMD_ADMIN pwd:init1234
SECSTORESLD SYNTAX
To execute this command you need to stop the agent. The command updates the credentials of the connection to the SLD system: Update the agent secstore file
smdsetup secstoresld user:slddsuser pwd:xxxxxx
CHANGESERVERNAME SYNTAX
To execute this command you need to stop the agent. The command updates the servername attribute of the agent: Update the agent configuration file
smdsetup changeservername servername:"myservername"
SUPPORTLOGS SYNTAX
This command retrieves all agent log files and executes some basics tests to check the agent pre-requisites as: jdk version check jvm parameters check hostname resolution check tcp port check p4 protocol check message server connectivity check Execute the command
smdsetup supportlogs
At the end of the execution of the command, a Zip file is generated in at the following location: /usr/sap/<AGENT_SID>/<AGENT_INSTANCE_NUMBER>/SMDAgent/supportLogs_<TIMESTAMP>.zip. The zip content is the following: Check results (AgentSupportToolReport.log in SMDAgent/log folder) Diagnostics Agent log files Diagnostics Agent profile SAPinst logs work directory content All properties files contains in SMD agent folder
SAP AG 2013
28
6.1.6 Change Agent log level

Main Diagnostics Agent log files are stored in folder: /usr/sap/<SID_AGT>/<InstNbr>/SMDAgent/log and organized as follow: Filename SMDSystem.log Default settings 10 files of 1MBytes Default level INFO Description This file reports all information about agent framework like connection to managing system, SLD report, agelets loading and patch sequence. This file reports all information about the agelets used by RCA applications, Setup Wizards and Wily IS setup. This file reports all information about agent connection layer with managing system.
SMDAgentApplication.log
10 files of 1Mbytes
WARNING
P4.log
10 files of 1MBytes
NONE
In case the Diagnostic Agent is currently connected to the Solution Manager System, the procedure to change the log level of these log location is: Go in Agent Administration UI Select the tab Agent Log viewer Select the relevant Agent in the dropdown list Configure the expected level. With Solution Manager 7.1, click on the link Log configuration to access the Agent log levels configuration UI. In general, avoid changing the log levels of all Diagnostics Agents at the same time (using button Save on All Agents). In case the Diagnostics agent is not connected to the Solution Manager System, the procedure to change the log level is: At OS level login as <Agent SID>adm OS user Stop the agent Go in folder /usr/sap/<SID_AGT>/<InstNbr>/SMDAgent/configuration Open the file log-configuration.xml Adapt the level (attribute effective-severity) in following line (level values: ALL,DEBUG,INFO,WARNING,ERROR,NONE): For Filename SMDSystem.log SMDAgentApplication.log P4.log
<log-controller effective-severity="INFO" maximum-severity="ALL" minimumseverity="DEBUG" name="/SMDlogger/System"> <log-controller effective-severity="WARNING" maximum-severity="ALL" minimumseverity="DEBUG" name="/SMDlogger/AgentApp"> <log-controller effective-severity="NONE" maximum-severity="ALL" minimumseverity="ALL" name="com.sap.engine.services.rmi_p4">
Save the file and restart the agent.
SAP AG 2013
29
6.2
Managing J2EE user
Information about the Solution Manager J2EE user needed by the Diagnostics Agent to connect to the Solution Manager System, is described: For Solution Manager 7.0 / 7.0 EHP1 in the User Administration guide, section 2.3.3 [SOLMAN.AGT.COM]: Diagnostics agent Communication User. For Solution Manager 7.1 in the Solution Manager Security Guide. Starting with Solution Manager 7.1 SP5, the Diagnostics Agent can connect to Solution manager server using an authentication via username / certificate, instead of using a basic authentication (user/password). Note that with the certificate authentication the user is not getting locked in case of multiple attempts (performed by several Diagnostics Agents). Pay attention to the following situation For Diagnostics Agents that have not been connected so far to a Solution Manager 7.1 SP05 system or higher, remind that the Agent will stop trying to connect/login on the Solution Manager system in case wrong user credentials have been provided (or the user is locked). In that context also be aware that in case the SMD_ADMIN user credentials are changed, offline Diagnostics Agent will not get notified of that change. At the time these Agents will go online again they will lock the SMD_ADMIN user, in case at least 3 or more Agents are in this situation. Since Solution Manager 7.1 SP05, consider using the Non-authenticated Agent list, as described by: http://wiki.sdn.sap.com/wiki/display/SMSETUP/Non-Authenticated+Diagnostics+Agents Most common agent connection problems Authentication failed all authentication problems are logged in file SMDSystem.log.
Exception: at com.sap.engine.core.thread.impl3.SingleThread.run (SingleThread.java:170) Caused by: com.sap.engine.services.security.exceptions.BaseLoginException: Authentication did not succeed. at com.sap.engine.services.security.login.ModulesProcessAction.run (ModulesProcessAction.java:175) ... 13 more
Before restarting the agent It is important to execute some checks about the J2EE user as follow: 1. Connect to the ABAP transaction SU01, to check if the J2EE user (SMD_ADMIN) is not locked and if the Role assigned is compliant to SAP recommendations done in the User Administration guide. 2. Logon to the Agent administration tool with the J2EE user (SMD_ADMIN) to check that the logon module of the Java stack works correctly. 3. Call the smdsetup to update only the credentials to be sure the password is up-to-date. 4. Restart the Diagnostic agent. In dev_smdagent.log, if you find the exit code -11112 it is a authentication problem as described above [Thr 880] Wed Feb 04 09:28:47 2009
[Thr 880] JLaunchIStartFunc: end of main method (rc = 0) [Thr 6000] JLaunchIExitJava: exit hook is called (rc = -11112) [Thr 6000] ************************************************************* *** ERROR => The Java VM terminated with a non-zero exit code. *** Please see SAP Note 943602 , section 'J2EE Engine exit codes' *** for additional information and trouble shooting. ********************************************************************** [Thr 6000] JLaunchCloseProgram: good bye (exitcode = -11112)
SAP AG 2013
30
Tracking Agent issues

7.1 OSS message pre-requisites
Before forwarding any OSS messages to the Dev support organization, it is important to check the following points (as far as concerning Diagnostics Agents): The Diagnostics agent has been installed with SAPinst 7.01 EHP1, 7.11, 7.20, 7.3x or Solution Manager 7.0 EHP1 SR1 or 7.1. The OS limit for Unix platform are compliant with the agent OS user (<AgentSID>adm) JDK vendor and version is supported by the Diagnostics Agent JVM parameters of Diagnostics Agent are compliant to recommendations The Diagnostics Agent connection to the Solution Manager system is of type J2EE Message Server connection The Solution Manager Java dispatcher is correctly sized to avoid Agent disconnections (see part 8.1 Solution Manager J2EE stack settings.) In case the check points cannot solve the problem, follow the SAP Note 1076573 to create the OSS message. Also in case the LM-SERVICE version is at least NW04S SPS17 Patch 3, attach the support logs Zip file generated by the smdsetup script (see part 6.1.5 Usage of smdsetup script).
7.2
Agent crash investigation
When the Diagnostics agent process shutdowns abnormally, the procedure to determine the cause is the following: At OS level login as <Agent SID>adm OS user Go in folder /usr/sap/<SID_AGT>/J<INSTANCE_NUMBER>/work Open the log file dev_jcontrol (This is the log file of the jcontrol process which is controlling the Diagnostics Agent java process) to check if the jcontrol has started the jlaunch process as described below:
[Thr 3288] JControlICheckProcessList: process SMDAgent started (PID:3616)
If the process jlaunch cannot be start, the error is probably located in SAP profile parameters
The following message is NOT a critical error:
JStartupICheckFrameworkPackage: can't find framework package C:\usr\sap\SMD\exe\jvmx.jar
The Diagnostics Agent does not use specific SAP kernel functions and this message can be ignored. The process jlaunch is started, open the file dev_SMDAgent which corresponds to the jlaunch process output :
[Thr 3860] JLaunchISetState: change state from [Initial (0)] to [Initial (0)] [Thr 3860] JLaunchISetState: change state from [Initial (0)] to [Waiting for start (1)] [Thr 3860] Thu Dec 25 12:11:35 2008 [Thr 3860] JLaunchISetState: change state from [Waiting for start (1)] to [Starting (2)] [Thr 3860] JLaunchISetState: change state from [Starting (2)] to [Running (3)]
In this file, all information about agent java process is stored like JVM loading, process state, crash of VM or bad JVM parameters. It is important to have a look this file to find out Exit Codes or Error messages.
JVM Loading problem
SAP AG 2013
End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide all JVM loading problems are logged in file dev_SMDAgent.
Error: [Thr 1084238144] *** ERROR => DlLoadLib: dlopen()= libjvm.so: cannot open shared object file: No such file or directory -> DLENOACCESS (0, Success) [dlux_mt.c 314] [Thr 1084238144] *** ERROR => Can't load VM shared library (libjvm.so) (rc=-2) [jhvmxx_mt.c 1576] [Thr 1084238144] *** ERROR => Cannot load DLL for Java VM [jlnchxxi_mt.717]
31
Here you can find potential JDK installation issues like : Wrong JDK version for Agent JDK is not compatible with SAP Kernel (32bits/64bits) Diagnostics Agent OS user have not rights to access to JDK Libraries The SAP Note 940893 - Analysis of errors within the startup of J2EE Engine 6.40 can help to troubleshoot JVM loading problems. If the JVM is loaded correctly, now check the file std_SMDAgent.out and jvm_SMDAgent.out to find some errors like OutOfMemory, StackOverflow, Nullpointer which stop the agent process.
SAP AG 2013
32
7.3
Thread dump
It is important to do thread dumps on the Diagnostics Agent process to have an understand of the context, in case a deadlock or not responding Diagnostic Agent situation is faced.
Make a Thread dump under Windows

The Diagnostics Agent is a SAP system and available in SAP MMC.
In SAP MMC, open the node AS Java Process Table of Diagnostics Agent identified by the SID and instance number. Only one process is displayed in the right part: It is the agent process. Select the process and right click on it to display the context menu, press the item Dump stack Trace.
Make a Thread dump under Unix For 7.0x

Logon in UNIX shell with the OS user of the Diagnostics Agent and execute the command smdadmin.sh XX (XX is the instance number of Diagnostics agent). For old agent installation, it is necessary to execute the command ../smdadmin.sh from the folder /usr/sap/<SID_AGT>/J<InstanceNumber>/script . Once the command is executed, the JCMON menu is displaying. Select the entry 20: Local administration menu. Then execute the action 10: Dump stacktrace on process 0 (This is the process of the SMDAgent).
For 7.1x, 7.2x and 7.3x

Logon in UNIX shell with the OS user of the Diagnostics agent and go to exe folder C:\usr\sap\DAA\SMDAXX\exe sapcontrol.exe -user <SID>adm <PWD> -nr <InstanceNumber> -function J2EEControlProcess SMDAgent DumpStackTrace example: /usr/sap/DAA/SMDA97.exe > ./sapcontrol -user smdadm abc123 -nr 98 -function J2EEControlProcess SMDAgent DumpStackTrace
Where is the thread dump stored?

Once the thread dump is done, the output is written in file /usr/sap/<SID_AGT>/J<InstanceNumber>/work/
std_SMDAgent.out with Sun and HP-UNIX JDK.
SAP AG 2013
33
For IBM JDK, the thread dump output is written in java.core.<timestamp>.txt in folder usr/sap/<SID_AGT>/J<InstanceNumber>/SMDAgent.To find the absolute path of an IBM thread dump, open the file std_SMDAgent.out and find the line beginning with Java Dump written to
7.4
Activate P4 logs
If a p4 connection problem (like timeout, connection reset) persists with the Solution Manager system or the Managed System after applying the different checks of this document, you should enable the P4 logs before sending the OSS message to the Development Support organization.
Connection problem to Managing System or Managed system

The procedure to enable the P4 logs on the Diagnostics agent is the following: Stop the agent Clean the log folder under SMDAgent directory Set the log level (effective-severity) of SMDSystem location to ALL (see 6.1.6 Change Agent log level) Set the log level (effective-severity) of P4 location to ALL (see 6.1.6 Change Agent log level) Add the Java parameter -Ddebug=true in agent configuration (6.1.2 Change JVM Parameters) Once the p4 log is enabled on the Agent, go to the Solution Manager system, or the Managed system to execute the following instructions: Open the Visual Administrator of Java stack Select the server node Under SID node Select the service Log Configurator Go in tab Locations Select the location com.sap.engine.services.rmi_p4 Increase the log level to DEBUG and save the modification. Once all P4 locations are configured, restart the agent to reproduce the problem and attach the agent logs (use the command smdsetup supportlogs) and the current defaultrace.log of the Managing or the Managed system to the OSS message. Performance Impact It is important to restore All log locations changed to their default level because the above described procedure has a high performance impact on the Diagnostics Agent and therefore indirectly on the Managed system or the Solution Manager system.
SAP AG 2013
34
Scalability
8.1 Solution Manager J2EE stack settings
SAP recommends adapting the J2EE configuration of Solution Manager Java stack especially in case you plan to connect over time more than 40 Diagnostics Agents to Solution Manager. The Java dispatcher has an important role in the Agent connection stability. Therefore, you need to adapt - using the Solution Manager system ConfigTool - the thread and connection settings, as well as the heap size configuration of the dispatcher node to avoid some timeout connection issues. Find below a summary of all parameters which have to be adapted. Also remind that a restart of the Solution Manager Java stack will be required. Dispatcher Parameter Max Heap Size (in MB) (JVM Parameter -Xmx) ThreadManager->MaxThreadCount ThreadManager->InitialThreadCount ThreadManager->MinThreadCount ConnectionManipulator ->maxSoTimeOutConnections Recommended Value Total number of Diagnostics Agents * 0.5 (Minimum 128MB) Total number of Diagnostics Agents * 2 + 50 Total number of Diagnostics Agents * 1.5 Total number of Diagnostics Agents Total number of Diagnostics Agents * 1.35 Example for 600 Agents 300 MB 1250 900 600 810
Default value of MaxThreadCount After an installation of Solution Manager, the default value of parameter MaxThreadCount in Java Dispatcher configuration is set to 70. These settings allow by default to connect a maximum of 40 Diagnostics Agents.
SAP AG 2013
35
Telnet SMD server commands
Some telnet commands are available in the Solution Manager J2EE to check the installation and configuration of the SMD server. Connect with telnet program to SAP J2EE server of Solution Manager (default port 5xx18) and go on server node with command jump <nodeid>. Add the group smdserver to enable the commands (use the command add smdserver). Once the group smdserver added, execute the command man g smdserver to have access to all available smd server commands. SMD server telnet command Smdstatus smdstatus a list_agents find_agent <servername> or <fqn> mm ON mm OFF kill_agent <servername> or <fqn> start_agent <servername> or <fqn> stop_agent <servername> or <fqn> host_mapping known_agents -c known_agents -n As Es Description Display the version and running mode of SMD server. Check each agent connectivity to detect some connection problem with one or several agents. Display all Diagnostics Agents connected to SMD server with their status. Retrieve an Agent using a servername, fully qualified domain name or IP. Enable the maintenance mode of SMD server Disable the maintenance mode of SMD server Reboot the Diagnostics Agent process Start the Agent applications of selected Diagnostics agent Stop the Agent applications of selected Diagnostics agent Display the mapping between servername and network names for all agents. Displays the list of known Agents. (Agents that have been connected at least once to the Solution Manager system) Displays the list of known Agents by FQN. Provides ping statistics for all Diagnostics Agents. Provide method statics for the Solution Manager server.
SAP AG 2013
36
10 Full Qualified Name resolution issue

The list of SAP notes following can help to solve some network resolution problems in different part of SAP products.
SAP Note Note 611361 - Hostnames of SAP servers Note 962955 - Use of virtual TCP/IP host names Note 773830 - FQHN determination in ICM Note 1282975 - Use of virtual TCP/IP host names in Windows Note 129997 - Hostname and IP address lookup Note 1052122 - Host names in SLD
Resume The use of hostnames that are too long or contain illegal characters may result in undetermined errors You want to use one or more logical host names for SAP software within an SAP server landscape to hide the physical network identities for the pieces of software from each other. It is not clear which fully qualified host name (FQHN) the Internet Communication Manager (ICM) is using. You want to use one or more logical host names for SAP software within an SAP server landscape to hide the physical network identities for the pieces of software from each other. Information about IP address lookup.
Note 979277 - Incorrect registration of comupter systems in the SLD
You wonder if a hostname that occurs in the System Landscape Directory (SLD) is correct. In the SLD, hostnames primarily occur in technical systems and destinations In computer systems registered by the ABAP data supplier in the SLD, attributes may be missing in certain cases (for example, details of the operating system).
SAP AG 2013

Diagnostics Agent Troubleshooting

Enviado por

Dados do documento

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Diagnostics Agent Troubleshooting

Enviado por

Direitos autorais:

Formatos disponíveis

SAP Solution Manager 7.0 EHP1, 7.

Diagnostics Agent Troubleshooting Guide

Type Style Example Text

Meaning Caution Example Note Recommendation Syntax

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

(also accessible via this tiny link http://wiki.sdn.sap.com/wiki/x/aQSAF)

Location of Intermediate Document

Old Document Location

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

Read SAP note 171356 before to install the agent

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

SLD production system

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

Agent installation strategy

Managed System environment

Managed Systems HA/Failover environment

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

E2E RCA configuration for Solution Manager 7.0x or higher

E2E RCA configuration for Solution Manager 7.1x only

<SIDADM USER> <SIDADM USER>

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

Information about Diagnostics Agents 7.11 or higher

Information about former Diagnostics Agents 7.0x

Exit code of smdstart.sh 0 2 5 4 3

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

su -c /usr/sap/SMD/J98/script/smdstop.sh > /dev/null 2>&1

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

About sapstartsrv process

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

59300 59301 59304 59306 59305

LB=1 LB=1 LB=1 LB=1 LB=1

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

LIST 0004105250 ncevhost.mydomain LIST 0004105200 ncevhost.mydomain

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

Connection configuration for managed systems

For the credentials : Connection Type to Managed system

Configuration stored in agelet

Connection Type to Managed system

Configuration stored in agelet

com.sap.smd.agent.application.global .configuration com.sap.smd.agent.application.global .configuration

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

6.1.1 Change Path of the JDK

For 7.00 and 7.01

For 7.1x , 7.2x and 7.3x

6.1.2 Change JVM Parameters

For 7.00 and 7.01

For 7.1x, 7.2x and 7.3x

6.1.3 Replacing JDKs 1.4.2 with SAP JVM 4

6.1.4 Verifying Logical hostname

For 7.01, 7.1x, 7.2x and 7.3x

6.1.5 Usage of smdsetup script

* smdsetup managingconf servername:"<...>"] * smdsetup managingconf servername:"<value>"]

hostname:"<fqn>" port:"<p4 port>" [optional user:"<value>" pwd:"<value>"

7.0 SP14 or higher

(In rare circumstances) To connect the agent with a P4 direct connection:

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide

Update the agent secstore file:

End-to-End Diagnostics Diagnostic Agent Troubleshooting Guide