Chapter 1 Overview of the NMS for iMonitor
1.6 Using the iMonitor Interface
1.6.5 Selecting Columns for Viewing
In any iMonitor pane with columns, any operator can select the set of columns that will be displayed whenever that operator views the pane. Once changed, the modified display will persist for that operator even after log off.
To select which columns to display:
1. Right-click anywhere in the column headings to display the column selection context menu.
Figure 33. Selecting Columns for Viewing from Context Menu
2. Select or clear individual columns for display from the menu one at a time or select More to view the Select Columns dialog box (Figure 34).
Figure 34. Select Columns Dialog Box
iBuilder User Guide 27 Using the iMonitor Interface
3. In the Select Columns dialog box, click the check boxes to select or clear the
corresponding columns for display. (Alternatively, select and clear a check box by first selecting the column name in the list, and then clicking the Show or Hide button.) Only selected columns are displayed in the pane.
4. Click Show All or Hide All to select or clear all check boxes.
Figure 35. Select Columns Dialog Box with Show Default Button 5. To change the order in which columns appear in the pane:
a. Click a column name to select it.
b. Click the Move Up button to move the selection one place up in the list. This moves the column to the left in the pane.
c. Click the Move Down button to move the selection down in the list. This moves the column to the right in the pane.
6. To change the width of a specific column, first select the column name. Then enter the new width in Width of selected column.
NOTE: In some cases, the Select Columns dialog box also contains a “Show Default” button (Figure 35). Click that button to select only the default columns.
iMonitor User Guide 29
2 Monitoring Conditions and Events
iMonitor displays both conditions and events. Conditions (alarms and warnings) indicate a problem or potential problem in an iDirect network. Events provide real-time information about the operation of the network but do not indicate a network problem. Table 3 identifies the types of notifications (conditions or events) iMonitor provides for each element in the Tree.
2.1 Conditions
Conditions in iMonitor are made up of Alarms and Warnings. Alarms indicate an interruption in service. Warnings indicate conditions that could result in an interruption of service if not handled in a timely fashion.
2.1.1 Representing the State of an Element with Icons
The icon representing a network element in the iMonitor Tree indicates the current state of that element. The network element states and the Icons that represent them are described in Table 4.
Table 3. Elements and Types of Information Provided
Elements Type of Incident
Table 4. Real-Time States and Icons
State Icon Meaning
OK The element is functioning properly. Shown in
order from left to right are a properly functioning PP, blade, line card, remote, and chassis.
OK This icon is seen in the Conditions Log and
indicates that the element is functioning properly.
Warning One or more Warning conditions is active for
the element.
Alarm One or more Alarm conditions are active for
the element (layer 2/3 alarm, unit not responding, etc.). Warnings may also be active in the Alarm state.
Mesh Alarm One or more Mesh Alarm conditions are active
for the element. (Not supported in iDX Release 3.3.)
Offline The remote has been sent offline.
Rx Only The remote is in receive-only mode
Elsewhere Indicates that a roaming remote is acquired in
a different network.
Sleep Mode The remote has entered sleep mode.
Unknown
Condition Condition state unknown
Disabled Disabled condition
iMonitor User Guide 31 Conditions
2.1.2 Conditions Pane
The current state of network elements is also represented in the Conditions Pane. Select View
Conditions to open that pane at the bottom of iMonitor’s main window.Figure 36. iMonitor Conditions Pane
The Conditions pane has tabs showing conditions using the following criteria:
• Active Conditions shows all outstanding conditions that have not been cleared. Any current alarms or warnings are displayed on this tab.
• Observation View shows all conditions for specific elements under observation. (See Placing an Element Under Observation on page 38.)
• Disabled Conditions shows all conditions that have been disabled. To disable an active condition, right-click the condition and select Disable Condition.
Figure 37. Disabling a Condition
• Condition Log shows the 500 most recent condition changes. Conditions on the Condition Log tab are sorted by the time that the condition change occurred by default.
To clear the contents of the Condition Log tab:
1. With the Condition Log tab selected, right-click anywhere in the Conditions Pane.
2. Select Clear List from the menu.
Figure 38. Clearing the Condition Log Tab
2.1.3 Elements with Multiple Conditions
Multiple conditions often exist simultaneously on a network element. In that case, the overall state of the element reflects the highest severity of any condition as follows:
• No conditions: overall state is OK
• One or more Warnings: overall state is Warning
• One or more Warnings and one or more Alarms: overall state is Alarm
• Remote has been sent Offline: overall state is Offline
2.1.4 Offline State
The offline state is a special condition that overrides all other warnings and alarms. This state applies only to remotes. The offline state can be initiated by a remote user before turning the remote off, to indicate to the network operator that no problem investigation is necessary.
When a remote is sent offline by the remote user, iMonitor and the NMS event server ignore all subsequent alarms. If a unit is turned off without sending it offline first, the remote enters the Alarm state at the hub.
The offline state clears automatically when the remote is turned back on and re-acquires the network.
2.1.5 Alarms and Warnings on Elements
Table 5 and Table 6 list the alarm and warning conditions that can be raised for the various elements.
Table 5. Explanation of Alarms by Element Element Alarm Condition Explanation
Chassis Chassis Down iMonitor cannot communicate with the chassis
Chassis Manager Down Chassis cannot communicate with NMS chassis manager process
Protocol Processor
Protocol Processor Down The heartbeat has not been received from the Protocol Processor
Hub Line Card
Line Card Down iMonitor cannot communicate with the Hub Line Card Rx SCPC Loopback C/N Line card downstream loopback exceeds clear sky C/N.
TDM Lock Hub line card is no longer locked to the downstream loopback
10 MHz Clock Alarm Board does not support 10 MHz clock GIG E Failed Gibabit Ethernet Port Failed
Hub Stats Alarm Stopped receiving statistics from hub line card Backplane Lost SOF Hub line card lost backplane Start of Frame NCR Missing NCR sync not received from Tx line card
PPS Missing Hub line card does not detect pulse per second signal
iMonitor User Guide 33 Conditions
Remote UCP out of Network Remote has time out on UCP message LAT Timeout Stopped receiving echo reply from remote
LL Down Remote Link Layer is down
Table 6. Explanation of Warnings by Element Element Warning Condition Explanation
Four Slot RCM A Not Present RCM A has not been installed in the Chassis Four Slot RCM A Fault RCM A of the chassis has failed
Four Slot RCM B Not Present RCM B has not been installed in the chassis Four Slot RCM B Fault RCM B of the chassis has failed
Four Slot Power Alarm A Bad Chassis power supply A has failed Four Slot Power Alarm A Over
Temperature
Chassis power supply A exceeds temperature limit
Four Slot Power Alarm B Bad Chassis power supply B has failed Four Slot Power Alarm B Over
Temperature Chassis power supply B exceeds temperature limit Four Slot FSM Not Present FSM has not been installed on the chassis
Four Slot FSM Fault FSM has failed on the chassis Four Slot FSM Fan Fault FSM fan has failed on the chassis Four Slot IFM Not Present IFM has not been installed on the chassis Four Slot IFM Fault IFM has failed on the chassis
Four Slot Alarm Disabled Audible alarms are disabled for the chassis Four Slot OPM A Fault OPM A has failed on the chassis
Four Slot OPM B Fault OPM B has failed on the chassis Hub Line
Card Rx Overflow of frames Downstream Packets per sec. overdrive
Back plane lost 10 MHz Clock The 10 MHz reference timing signal is absent from the chassis backplane
Acq Hub CRC Errors Number of acquisition CRC errors exceeds maximum Table 5. Explanation of Alarms by Element
Element Alarm Condition Explanation
(Hub Line
Card) Traffic Hub CRC errors Number of traffic CRC errors exceeds maximum Fll DAC Errors Internal oscillator is out of calibration. (Also referred
to as Clock DAC)
Downstream PPS overdrive Downstream overdrive condition detected on line card
GIGE health Error on GIGE Ethernet port of the line card Inroute AGC out or range AGC measurement above or below defined limits Loss of NCR Sync NCR count not synchronized to Master NCR count Protocol
Processor Blade CPU high Blade CPU usage is above the defined limit Remote AGC Out of Range Remote Automatic Gain Control outside limits.
Blade no Encryption License The remote’s PP blade has no encryption license Calibrated Transmit Power Transmit power is outside the defined limits Certificate has Expired X.509 certificate on TRANSEC remote has expired Certificate is Invalid X.509 certificate on TRANSEC remote is invalid Certificate will Expire X.509 certificate on TRANSEC remote will expire in 30
days
CRC8 Errors Number of CRC8 errors exceeds limit CRC32 Errors Number of CRC32 errors exceeds limit
Downstream SNR out of range The downstream C/N measured at the remote is outside limits
Fan Alarm An internal fan has failed. Raised for 73xx remotes only
Fll DAC Errors Internal oscillator is out of calibration. (Also referred to as Clock DAC.)
GPS Signal Lost Mobile remote has lost GPS input
Latency Measured latency, hub to remote, exceeds the define maximum
Local LAN Disconnect LAN port on remote is disconnected
Lost Contact PP has temporarily lost contact with remote
Maximum Tx Power Remote transmit power is approaching the maximum Nominal Carrier Equivocation
Occurring
A remote is switching nominal carriers rapidly on a specific inroute ID
Timing Offset The remote’s timing offset is out of range Remote Off-line State The remote has been sent off-line
Remote Temperature Remote temperature is outside the defined limits Table 6. Explanation of Warnings by Element (continued)
Element Warning Condition Explanation
iMonitor User Guide 35 Conditions
2.1.6 NMS Database Replication Warnings
Beginning with iDX Release 3.1, iDirect supports replication of NMS databases from the Primary NMS Server (MySQL Master) to one or more Backup NMS Servers (MySQL Slaves). For a detailed description of this feature, see the chapter titled “NMS Database Replication” in the Technical Reference Guide.
The scripts that make up the iDirect replication tool set send warnings and events to the NMS Event Server for display in iMonitor. For example, on successful completion of any task each script sends an event listing the task and the fact that it was completed successfully. On any task failure each script sends a warning listing the task and a brief description of the failure.
Table 7 lists the iMonitor warnings that can be raised by the NMS Database Replication scripts that run on the Primary NMS Server (MySQL Master).
Remote Tx Acq CRC Number of TDMA upstream acquisition CRC errors exceeds limit
Remote Tx Data CRC Number of TDMA upstream data CRC errors exceeds limit
Remote Tx Acq Mismatch Number of TDMA upstream acquisition bursts from an unexpected remote exceeds limit
Remote Tx Data Mismatch Number of TDMA upstream data bursts from an unexpected remote exceeds limit
Remote Tx Data Missing Number of TDMA upstream data slots for which no data was received exceeds limit
Remote Tx SCPC HDLC Error Number of upstream octet and CRC errors exceeds limit for SCPC return channel
Remote Tx SCPC Data Mismatch Number of upstream IP packets from an unexpected remote exceeds limit for SCPC return channel Rx SCPC C/N Remote receive downstream carrier exceeds clear sky
C/N. Value will be -1 or -100.
Upstream C/N0 out of range The remote’s upstream C/N0 measured at the hub is outside limits
NOTE: If NMS Database Replication fails, an active condition will be raised in iMonitor. It is important to take action to recover from the failure and clear the condition. If no action is taken, the replication log files on the Primary NMS Server can no longer be purged and it is possible to run out of disk space. For more information, see the Technical Reference Guide.
Table 6. Explanation of Warnings by Element (continued) Element Warning Condition Explanation
Table 7. Warnings Raised by NMS Database Replication Scripts on MySQL Master
Script Name Warning
cr8DbMaster Specified slave does not appear to be an IP address, aborting.
Specified slave matches master hostname, aborting.
Specified slave matches master IP, aborting.
Specified slave is not reachable, aborting.
User “idirect” cannot “ssh” into host specified slave without a password, aborting.
iDirect NMS Servers RPM is not installed, aborting.
MySQL is not installed, aborting.
MySQL account info missing, user: admin.
MySQL account password missing, user: admin.
MySQL account info missing, user: app.
MySQL account password missing, user: app
The database contains DB tables that are not supported in this version of the iDX product, aborting.
Failed to connect to MySQL database on master with credentials: <db user>, <db password>.
MySQL is not installed on slave, aborting.
MySQL version on slave does not match MySQL version on master, aborting.
NMS servers not installed on slave <slave>, aborting.
NMS servers version on slave does not match version on master, aborting.
Existing slave server specified. Use “-f” to force re-creation of an existing slave server.
iMonitor User Guide 37 Conditions
Table 8 lists the iMonitor warnings that can be raised by the NMS Database Replication scripts that run on a Backup NMS Server (MySQL Slave).
mysql-repl-monitor Replication error detected on slave.
iDirect NMS Servers RPM is not installed, aborting.
MySQL is not installed, aborting.
Slave is not reachable, aborting.
MySQL is not installed on slave, aborting.
MySQL version on slave does not match MySQL version on master, aborting.
NMS servers is not installed on slave, aborting.
NMS servers version on slave does not match MySQL version on master, aborting.
Failed to connect to MySQL database on slave repl-file-send Host is not reachable, skipping.
User “idirect” cannot “ssh” into host “<slave_ip>” without a password, skipping.
FAILED while writing files to slave: <slave_ip>.
MySQL account info missing, user: admin.
MySQL account password missing, user: admin.
MySQL account info missing, user: app.
MySQL account password missing, user: app
Failed to connect to MySQL database on master with credentials.
Table 8. Warnings Raised by NMS Database Replication Scripts on MySQL Slave
Script Name Warning
cr8DbSlave MySQL account info missing, user: admin.
MySQL account password missing, user: admin.
MySQL account info missing, user: app.
MySQL account password missing, user: app.
Table 7. Warnings Raised by NMS Database Replication Scripts on MySQL Master (continued)
Script Name Warning
2.2 Placing an Element Under Observation
An element can be placed “under observation” in order to monitor its conditions. The following elements can be placed under observation:
• Protocol Processor
• Blade
• Line Card
• Remote
• Chassis
To place an element under observation:
1. Right-click the element in the iMonitor Tree and select Under Observation from the menu.
2. Select View
Conditions from the main menu or click the Toggle Conditions icon (eyeglasses) on the main toolbar.iMonitor User Guide 39 Placing an Element Under Observation
Figure 39. Selecting the Toggle Conditions Icon
3. Click the Observation View tab to view only the conditions for elements under observation.
Figure 40. Observing Conditions
4. To cancel observation of an element, right-click a condition and select Cancel Observation.
Figure 41. Cancelling Observation of an Element
NOTE: Canceling the observation of an element removes that element’s conditions from the Observation View.
5. To view the control panel (Figure 42) for a remote under observation, right-click a remote condition and select Remote Control Panel or double-click the remote condition. See Control Panel on page 125 for details.
Figure 42. Remote Control Panel
2.2.1 Viewing Conditions or Events
It can be useful to retrieve events over an extended time period for one or more remotes.
Although it is possible to retrieve all events and sort the results, iMonitor can also filter historical events. Specify a text filter to display only those events that match the filter.
The text filter is available at the bottom of the historical time range parameters dialog box (Figure 45 on page 42). A text filter can be entered either prior to retrieving events or from the Time Range button on an existing events display. The filter values are applied only to the Event Description section of the event message.
The simplest filter string is a substring of the event description, such as “server”. Any event message that contains the specified substring is displayed in the results. The text filter field also supports Linux regular expressions for complex search filters. For instructions on forming regular expressions, see any Linux reference book.
iMonitor User Guide 41 Placing an Element Under Observation
To retrieve and view conditions or events for an element:
1. Right-click the element in the tree and select Conditions or Events to open the Select Items dialog box.
Figure 43. Select Items Dialog Box 2. To retrieve conditions or events in real-time, click OK.
3. To view past conditions or events, click either Historical or Get Past.
a. For Historical data, click Time Range to open the Select Time Range dialog box (Figure 44 for Conditions and Figure 45 for Events). Alternatively, click the ellipses (...) next to the Start Time and End Time to set the time range using the graphical clock.
NOTE: A Text Filter in the Get Past time range dialog box (Figure 45) only applies to the Event Description field of the event message.
Figure 44. Conditions Select Time Range Dialog Box
Figure 45. Events Time Range Dialog Box with Text Filter b. For Get Past, select a past time interval from the Get Past drop-down list.
iMonitor User Guide 43 Placing an Element Under Observation
Figure 46. Get Past Drop-Down List
4. Select the elements for which to view conditions or events. The options in the Select Items dialog box differ depending on the level of the element selected in the tree.
5. Click OK to view the conditions or events. For conditions, follow the directions beginning in Step 6. For events, see Step 10 on page 45.
6. When retrieving conditions, the Conditions pane opens showing the conditions logged over the specified time period in a multicolumn format (Figure 47).
Figure 47. Conditions Results in Multicolumn Format
7. To view conditions in a graphical format, click the Time Line tab (Figure 48).
Figure 48. Conditions Time Line Results in Graphical Format
NOTE: On the Conditions tab, a remote may have an arrow next to it. Click the arrow to expand the view to show all conditions for that remote.