Customers using IBM Tivoli Monitoring products or the components of Tivoli Management Services can encounter problems such as missing workspaces or historical data, or a reflex automation script that does not run when it should. In many cases you can recover from these problems by following a few steps.
Note: Use the trace settings indicated in these troubleshooting instructions only while you are trying to diagnose a specific problem. To avoid generating excessive trace data, go back to the default trace settings as soon as the problem is solved.
About the tools
You can access several troubleshooting tools, such as the Log analyzer or pdcollect tool to help you troubleshoot your IBM Tivoli Monitoring product or the
components of Tivoli Management Services.
ITMSuper Tools
The ITMSUPER Tools give you information about the health of your managed systems, situations, and environment configuration. You can find the tools by searching for “ITMSUPER” in the IBM Integrated Service Management Library (http://www.ibm.com/software/brandcatalog/ ismlibrary).
pdcollect tool
The pdcollect tool collects the most commonly used information from a system. It gathers log files, configuration information, version information, and other data. You can also use this tool to manage the size of trace data repositories. For more information see “pdcollect tool” on page 68.
IBM Support Assistant
The IBM Support Assistant is a free, stand-alone application that you can install on any workstation. Then, you can enhance the application by installing product-specific plug-in modules for the IBM products you use. For more information see “Support information” on page 335.
I am trying to find out what software is supported
Use resources in the IBM Tivoli Monitoring Installation and Setup Guideand the IBM website to determine the software that is supported. This enables you to find platform or database information for specific products.
The following resources are available to determine the software that is supported:
v For specific information about the supported software for IBM Tivoli
Monitoring, see “Hardware and software requirements” in theIBM Tivoli Monitoring Installation and Setup Guide
v For platform and database support information for most Tivoli products, consult
the matrix at Tivoli Supported Platforms (http://www-306.ibm.com/software/ sysmgmt/products/support/Tivoli_Supported_Platforms.html)
Workspaces are missing or views are empty
You can encounter a problem that Tivoli Enterprise Portal workspaces are missing or views are empty. For example, you may have workspaces that return no data. Symptoms of the problem:
v The workspaces return no data.
v There are no child Navigator items under the agent node in the Navigator view.
See “Resolving application support problems” on page 13.
v The Navigator items are labeled with internal names, such asKnt:KNT1076
instead of the correct names (such as Disk). See “Resolving application support problems” on page 13.
v You receive message KFWITM217E: Request error: SQL1_CreateRequest failed,
rc=209. See “Resolving application support problems” on page 13.
v You receive message KFWITM220E: Request failed during execution.See
“Resolving monitoring agent problems” on page 16.
For more information on workspaces that relate to historical data, see “Historical data is missing or incorrect” on page 22.
To diagnose the problem that workspaces are missing or empty, see “Diagnosing that workspaces are missing or empty.”
Diagnosing that workspaces are missing or empty
You can diagnose that workspaces are missing or empty by verifying that the monitoring agent has been started and that the configuration is correct. You can also check that application support has been added.
About this task
To diagnose that workspaces are missing or empty, perform the following steps:
Procedure
Preliminary diagnostics
1. Refresh the Navigator by clickingView>Refresh.
2. Verify that the monitoring agent has been started. Restart if necessary. In the Tivoli Enterprise Portal, right-click the Navigator item of the monitoring agent and click Start or Restart
3. Verify that the monitoring agent configuration is correct.
4. If your data is missing in an Oracle Agent workspace, see “Resolving Oracle DB Agent problems - diagnostic actions” on page 34. Similar problems might exist for other monitoring agents.
5. Check that application support has been added. See “Resolving application support problems” on page 13.
What to do next
For more information on actions that relate to these diagnostics, see the problem resolution tasks.
Resolving application support problems
Application support problems are caused by a lack of application support or an application support level mismatch among the components: monitoring server, portal server, desktop and Java™Web Start clients, and monitoring agents. Check the installed level of application support or run the ITMSUPER Tivoli Enterprise Monitoring Server analysis tool (or both) to get more information.
Before you begin
Complete one or both of the following tasks to ensure that this is an application support problem:
v “Diagnosing that workspaces are missing or empty” on page 12 v “Diagnosing that a situation does not raise when expected” on page 27
About this task
To resolve application support problems, you perform diagnostic and corrective actions. These actions include checking application support on the servers and client and running the Tivoli Enterprise Monitoring Server tool to ensure that application support is installed consistently in your environment.
Procedure
Diagnostic and corrective actions
1. Check application support on the monitoring server, portal server, and portal client:
v Run the kincinfo.exe -icommand in the%CANDLE_HOME\InstallITM
directory to show what is installed.
v Run the./cinfo –icommand in the$CANDLEHOME/bin directory to
show what is installed.
v (monitoring server) Look in the&rhilev.&rte.RKANDATV data set where
&rhilev is the high-level qualifier and &rte is the mid-level qualifier of the libraries for the runtime environment where the monitoring server is configured for files namedKppCATand KppATRwhereppis the two-character product or component code.
2. You can also run the Tivoli Enterprise Monitoring Server analysis tool provided by ITMSUPER against the hub monitoring server to ensure that application support is installed consistently throughout your environment.
3. If application support is missing, add the appropriate application support to the portal server and monitoring server for the monitoring agents.
4. If the desktop client or Java Web Start client is being used, ensure application support is installed on the portal client.
What to do next
For more information and instructions on installing application support see
“Configuring application support for nonbase monitoring agents” in theIBM Tivoli
support on a z/OS monitoring server, see “Adding application support to a monitoring server on z/OS” in Configuring the Tivoli Enterprise Monitoring Server on z/OS.
Resolving monitoring server problems
Monitoring server problems are caused by a monitoring server that is not started or connectivity that is lost either between servers or between servers and agents. You can restart the Tivoli Enterprise Monitoring Server, and you can also run the ITMSUPER Topology tool to get more information.
About this task
To resolve monitoring server problems, you perform diagnostic and corrective actions. These actions include running tools, such as the Topology or Connectivity tool and correcting communication failures in logs.
Procedure
Diagnostic and corrective actions
1. If you are an administrator, restart the monitoring server. Otherwise, notify an administrator and wait for the monitoring server to be restarted.
2. Running the following ITMSUPER tools might also provide more information:
v Topology tool v Connectivity tool
v Tivoli Enterprise Monitoring Server analysis tool v Tivoli Enterprise Portal Server
3. Check the portal server logs for messages indicating communication failures to the monitoring server.
4. Check the monitoring server logs for messages indicating communication failures to the remote monitoring servers or to monitoring agents.
5. Correct the communication failures indicated in the logs.
Resolving monitoring agent problems
If the monitoring agent is running but data is not being returned or if you receive an error message from an agent log, such as Endpoint unresponsive, verify that the agent is connected and online. You can also verify that application support has been installed correctly.
About this task
To resolve monitoring agent problems, you perform diagnostic and corrective actions. These actions include verifying that the agent is running and that application support has been installed correctly. For information on monitoring agents on z/OS® see each product'sProblem Determination Guide.
Procedure
Diagnostic and corrective actions
1. Verify that the agent is connected. Check the monitoring server log for messages similar to Remote node <SYS:MQIRA> is ON-LINE.
2. If the agent is online, check to see whether subnodes are online in the agent log. For example: KMQMI171I Node JSG1:SYS:MQESA is online.
3. If subnodes are online, are workspaces showing correct titles?
v No: Verify that application support has been installed correctly and that
buildpresentation.batran correctly.
v Yes: Go to the next step.
4. If workspaces contains titles, is there a column heading?
v No: Verify that application support has been installed correctly and that
buildpresentation.batran correctly.
v Yes: Go to the next step.
5. If there is only a column heading with no data, turn on UNIT:KRA ALLin the agent and verify that rows are being returned when the workspaces are displayed.
Status of a monitoring agent is mismatched between the portal client
and tacmd command
You can encounter a problem that the status of a monitoring agent is mismatched between the Tivoli Enterprise Portal client andtacmd command. For example, a monitoring agent shows as online in the portal client and offline in the results of a
tacmd command.
Diagnosing that the status of a monitoring agent is
mismatched between the portal client and tacmd command
You can diagnose that the status of a monitoring agent is mismatched between the Tivoli Enterprise Portal and tacmd command by setting a trace to determine whether the problem is in the Tivoli Enterprise Portal Server or the Tivoli Enterprise Monitoring Server.
About this task
To diagnose that the status of a monitoring agent is mismatched between the portal client and tacmd command, perform the following steps:
Procedure
Preliminary diagnostics
1. Verify the state of the monitoring agent in Manage Tivoli Monitoring Services.
2. Compare the status of the node in the physical Navigator view with the status reported in the Managed System Status workspace. If the status in the physical Navigator view agrees with the status shown in the Managed System Status workspace, then the problem is at the monitoring agent. See “Resolving monitoring agent problems” on page 16.
3. To determine whether the problem is in the portal server or monitoring server, set the following trace in the portal server: ERROR (UNIT:ctcmw IN ER)./
4. Then examine the portal server log for the following statement:Node Status event received (managed system name).
v If the trace shows that the last node status record received for the managed
system matches the status shown in the portal client, then the problem is located in the monitoring server. See
v If the trace shows that the last node status record received for the managed
system indicated the correct status, then the problem is located in the portal server. Run the portal server trace, collect logs, and call IBMSoftware Support.
What to do next
For more information on actions that relate to these diagnostics, see the problem resolution tasks.
Resolving monitoring agent problems
Monitoring agent problems, such as a monitoring agent that has not started can be resolved by refreshing the Navigator status in the Managed System Status
workspace.
About this task
To resolve monitoring agent problems, you perform diagnostic and corrective actions. These actions include checking the status of the monitoring agent and the monitoring server.
Procedure
Diagnostic and corrective actions
1. Open the Managed System Status workspace and clickView>Refresh.
2. Make sure the monitoring agent is connected to the correct monitoring server.
3. Check the status of the monitoring server that the monitoring agent is connected to. For more information, see the monitoring server problem resolution task.
Resolving monitoring server problems
Tivoli Enterprise Monitoring Server problems, such as loss of connectivity between a monitoring agent and a remote monitoring server can be resolved by checking, for example, that the remote monitoring server is connected to the hub monitoring server.
Some causes of monitoring server problems:
v A remote monitoring server has shut down
v Loss of connectivity between the monitoring agent and the remote monitoring
server to which it reports, or between that monitoring server and the hub monitoring server
v You receive the following message in the monitoring server log: KDS9151E: The heartbeat from remote TEMS variable
was not received at its scheduled time and the remote TEMS has been marked offline.
About this task
To resolve monitoring server problems, you perform diagnostic and corrective actions. These actions include running tools, such as the ITMSUPER tools and correcting connectivity failures.
Procedure
Diagnostic and corrective actions
2. If the monitoring agent is connected through a remote monitoring server, confirm that the remote monitoring server is connected to the hub monitoring server.
3. If the remote monitoring server is not running and you are an administrator, restart it. Otherwise, notify an administrator and wait for the remote
monitoring server to be restarted.
4. Running the following ITMSUPER tools might also provide more information:
v Topology tool v Connectivity tool
v Agent response time tool
v Tivoli Enterprise Monitoring Server analysis tool 5. Correct the connectivity failures identified.
The portal server does not start or stops responding
You can encounter a problem that the Tivoli Enterprise Portal Server does not start or stops responding. For example, you may receive a message that communication with the portal server could not be established or the portal server is not ready. Symptoms of the problem
v Portal client logon fails. “Diagnosing that portal server logon fails” on page 20. v The portal server stops responding during normal operation of the portal client. v You receive message KFWITM091E: View not available at this time.
v You receive message KFWITM010I: Tivoli Enterprise Portal Server not ready. v You receive message KFWITM402E: Communication with the Tivoli Enterprise
Portal Server could not be established.
v You find a similar text string toKFWDBVER, version not found when trying to
start the portal server. See “Resolving database problems - missing table or portal server database” on page 18.
Diagnosing that the portal server does not start or stops
responding
You can diagnose that the Tivoli Enterprise Portal Server does not start or stops responding by running the Tivoli Enterprise Portal Server Analysis ITMSUPER tool.
About this task
To diagnose that the portal server does not start or stops responding, perform the following steps:
Procedure
Preliminary diagnostics
1. For more information about any messages received, see theIBM Tivoli
Monitoring Messages(http://pic.dhe.ibm.com/infocenter/tivihelp/v61r1/topic/
com.ibm.itm.doc_6.3/messages/itm_messages.htm) reference guide. Operator responses and general information are provided for each message.
2. Allow the portal client enough time to establish a connection with the portal server.
v Yes: See step 4.
v No: See “Resolving database problems - instance not started” on page 19. 4. Collect the portal server log or the operations log and look for the following
text strings:
v KFWDBVER, version not found v TEPS database not found v user ID or password invalid v DB2 instance not started
5. Run the Tivoli Enterprise Portal Server Analysis ITMSUPER tool.
What to do next
For more information on actions that relate to these diagnostics, see the problem resolution tasks.
Resolving database problems - missing table or portal server
database
Database problems caused by a missing table or Tivoli Enterprise Portal Server database, or by a mismatch between the portal server version and the version record in the database, can be resolved by reconfiguring the portal server.
About this task
To resolve database problems, you perform diagnostic and corrective actions. These actions include reconfiguring the portal server.
Procedure
Diagnostic and corrective actions
1. To reconfigure the portal server, open Manage Tivoli Monitoring Services, right-click the portal server, and selectReconfigure. If the problem persists, run one of the following commands and set the correct password in the window that is displayed:
v For an SQL database,cnpsdatasource.exe v For a DB2 database,db2datasource.exe
2. To reconfigure the portal server, take one of the following steps:
v On the GUI interface, open Manage Tivoli Monitoring Services right-click the
portal server, and select Reconfigure.
v On the command-line interface, run the ./itmcmd config -A cq command. 3. Run the buildpresentationscript.
Resolving database problems - user ID and password
Database problems caused by a password that does not match the operating system password or an incorrect password in the registry can be resolved by reconfiguring the Tivoli Enterprise Portal Server and verifying the user ID and password.
The causes of database problems include:
v Portal server database user password is out of sync.
v Password does not match the operating system password. v Registry does not have the correct password.
About this task
To resolve database problems, you perform diagnostic and corrective actions. These actions include reconfiguring the portal server and ensuring that your portal client user ID is the same as the logon user ID of your system.
Procedure
Diagnostic and corrective actions
v To reconfigure the portal server, run thetacmd configureportalserver
command. If the problem persists, take one or more of the following steps: – Ensure that your portal client user ID is identical to the logon user ID of your
system and use the correct capitalization for your user ID and password. If you need to change your password, take the following steps:
1. Right-clickMy Computerand selectManage.
2. SelectLocal Users and Groups.
3. SelectUsers.
4. Right-click your user ID and selectProperties.
5. For db2admin, set the password to never expire.
– Check theDB2 UDB database and ensure that the db2admin user ID and password match those of the db2admin local account:
1. ClickControl Panel>Administrative Tools>Services.
2. Right-clickDB2 - DB2and selectProperties.
3. Select theLog Ontab and ensure that the db2admin user ID and