Operation Error Management

(1)

Operation — Error Management

• Monitors performance of SpeedNet Radio Ethernet and wireless interfaces • Provides consolidated performance reports on a per radio subgroup level • Supports continuous monitoring and management of radio configuration • Supports continuous monitoring and management of network topology • Supports scheduled software upgrade and backup for one or more radios • Supports periodic configuration backups

SpeedNet Radios enable utilities to build wireless mesh networks for small area coverage such as an electrical distribution feeder, and networks can be expanded to cover an entire metropolitan area. The SpeedNet system provides fully integrated, two-way communication of real-time information between smart grid components over a peer-to-peer mesh network. SpeedNet Radios automatically build routing tables that define message paths so, if a link is lost, a new path will be quickly established. Oper-ating in the unlicensed 902- to 928-MHz band, typical range with an omnidirectional antenna is 2 to 5 miles in mesh mode, and more than 6 miles in point-to-multipoint mode with directional antennas. If needed, SpeedNet Repeater Radios can be installed to increase transmission range and provide alternate nodes for message routing. See Figure 1.

(3)

Figure 2. IntelliTeam CNMS main menu.

The modular sections of the IntelliTeam CNMS system are opened from the Main menu accessed with the Start button. See Figure 2. A hierarchy of menus and

(4)

Error management is a set of functions that: • Detect malfunctions in a network • Maintain and examine error logs

• Accept and act on error-detection notifications

Error management provides the means to identify issues and coordinate resolution by assigning alarms to other users. To enable this, this module is organized as described below:

• Logging of Faults and Log Management—All errors are recorded in the error log

with a timestamp. This log can be viewed using the IntelliTeam CNMS Log Browser. In addition to managing the view list for user-friendly views, the log can also be purged and archived.

• Alarm Management—Features such as assignment of alarm to specific users; alarm

filtering; alarm forwarding; and housekeeping by deleting, purging, and archiving old alarms, are used for effective management.

• Tools to manage the alarms—Alarm SNMP notification and automatic email

notification are some of the actions triggered by an alarm. The IntelliTeam CNMS sys-tem enables configurations so these actions are only triggered for a filtered alarm list.

Alarms

The IntelliTeam CNMS system generates an alarm when an event indicates a failure. Set of events representing a common error or failure are correlated into alarms. For all events with the same failure object, one alert is generated, and the severity is set as the severity of the latest event with this failure object. The generated alarm is communicated to the user as an email. It is also stored in the database.

Whenever a new event is generated, the application ensures that there is only one alarm in the IntelliTeam CNMS database for a specific root cause. Since the reasons for problems differ across networks and depend on the traps and notifications supported, The IntelliTeam CNMS system offers a configurable list of fields that can be marked as unique for generating or updating an alarm. By default, it is configured to create an alarm for each unique combination of network elements, source (for example: Port-3-2) and category (For example: interface failure). This consolidation process helps in handling event bursts when a network element raises the same trap repeatedly. To meet this criterion, the IntelliTeam CNMS system takes the following actions:

Generation of a new alarm—A new alarm is generated when there is no alarm in the

IntelliTeam CNMS database for the configured combination (Default criteria source, category and network element combination).

Updating an existing alarm—Whenever an alarm is generated for a device for a specific

condition for which an alarm is already associated with it, the new event log updates the old alarm with the details instead of creating a new alarm. However, the related event logs enumerate the number of times this alarm was reported.

The alarm browser provides users with a summary of the alarms currently active in the network elements. The following functions are available from the alarm browser: • View the details of an alarm

• Filter alarm details to see the desired alarms

• Manage the alarms by assigning the alarms to various users • Acknowledge one or more alarms

• Clear one or more alarms manually

(5)

• Purge and archive the alarms

• Forward one or more alarms to the northbound clients • Export the selected alarms to a desired file format • Print the selected/filtered alarms

Viewing Alarms

The Alarm Browser window gives a summary of the alarms currently active in the network elements. It displays the alarm-specific details in a tabular view indicating information on network element, the affected component, the reason for the alarm, the severity of the alarm, and similar parameters. A view of the list of alarms for a group of network elements is also available under the Alarms tab at the Group level in the tree.

To view the alarm browser window:

1. Navigate to the Alarms tab window in the Application landing window.

2. The Alarm Browser window displays the list of alarms along with the alarm details. See Figure 3 on page 6. The details displayed carry the following information:

Annotation Indicator—An icon that shows whether a user has made an annotation

to the alarm.

Remedy Indicator—An icon that shows whether a users has noted a remedy to the

alarm.

Alarm ID—The unique ID of the alarm. Severity—The severity of the problem.

Element Name—The name of the radio reporting the problem. Subgroup—The subgroup of the radio reporting the problem.

IP Address—The wireless IP address of the radio reporting the problem. Message—A message/fault description obtained from the device. Date Updated—The date of last update of the alarm.

Acknowledgement Status—The field to indicate whether the alarm is acknowledged

or not.

Owner—The owner of the alarm.

Source—The component of the network element having the fault. Category—The alarm category such as Ac Power, Availability. Event Type—Threshold, CNMS, status polling, data collection.

Trigger—Indicates whether the current state of the alarm was caused by a failure,

conflict, success, or resolution.

Date Created Time—The date on which alarm raised.

Date Acknowledged—The date of acknowledgement, if the alarm is acknowledged. Cleared Time—The cleared time, if the alarm is already cleared.

(6)

Figure 3. Alarm Browser window.

Editing Alarm Details

The view displayed by the Alarm Browser can be edited to: • Acknowledge one or more alarms

• Provide comments for future reference • Change the owner for the alarm

These changes can be saved to maintain a history of change. To edit the alarm details:

1. Navigate to the Alarms tab on the Application landing window. 2. Select an alarm from the Alarm Browser window. See Figure 3.

3. Click Alarm Details icon , or right click on the selected alarm and select Alarm

Details. The Alarm Details window opens. See Figure 4 on page 7.

4. Click the Acknowledge check box to acknowledge the alarm. The History pane displays history of the selected alarm.

Note: An alarm can be acknowledged only by its owner.

5. Enter the required Remedy and Annotation. The user ID and the date/time of annota-tion are automatically recorded.

6. Click Update to save the changes.

7. Click on the Event Details tab to view the Events window. See Figure 5 on page 7.

(7)

Figure 4. Alarm Details window.

(8)

Acknowledging Alarm

To acknowledge an alarm:

1. Navigate to the Alarms tab on the Application Landing window.

2. Select any alarm other than a cleared alarm from the Alarm Browser window. See Figure 3 on page 6.

3. Click on the Acknowledge icon ,or right click on the selected alarm and select the Acknowledge menu.

4. Change the Acknowledged Status from N to Y.

5. The alarm can also be acknowledged from the Alarm Details window. See Figure 4 on page 7.

6. Select the Acknowledge check box to acknowledge an alarm. Then click Update.

Clearing Alarm

Whenever an error is corrected in a network element, the network element generates a

Clear alarm, which clears the alarm automatically. While users can clear an alarm gener-ated by the CNMS server after acknowledging it, they cannot manually clear an alarm generated by a network element.

Whenever a CNMS alarm is cleared, the user is asked to enter the reason for clearing the alarm. This is added as an annotation to the alarm. When the alarm is cleared, its severity value is changed to Cleared, and its severity color changes accordingly. The Related Alarm Logs tab gives details on the history of any alarm, giving informa-tion on when the alarm was raised and when it was cleared.

To clear an alarm:

1. Navigate to the Alarms tab window in the Application Landing page. Select a CNMS alarm from the list.

Note: Only acknowledged alarms can be cleared.

2. Click on the Clear icon , or right click on the selected alarm and select Clear menu. 3. Click OK in the confirmation window to proceed, or click Cancel to abort the Clear

process.

4. The Annotation window opens. Enter the required remarks and click Save.

5. The cleared alarm is added in the cleared alarms count and is listed in green above the tool bar.

Acknowledging and

Clearing Alarms

Alarms can now be acknowledged and cleared with a single button. To acknowledge and clear an alarm:

1. Navigate to the Alarms tab. Select a CNMS alarm from the list.

2. Click on the Acknowledge & Clear icon , or right click on the selected alarm and select “Acknowledge & Clear.”

3. Click OK in the confirmation window to proceed, or click Cancel to abort the

Acknowledge & Clear Process.

4. The Annotation window opens. Enter the required remarks and click Save.

(9)

Assigning Owners

Users with sufficient privilege can assign alarms to other users. This feature can be used by the administrator to assign alarms to network operators, so they can review and resolve a problem. To assign an owner:

1. Navigate to the Alarms tab in the Application Landing page. 2. Select an alarm from the Alarm List window.

3. Click on the Assign Owner icon , or right click on the selected alarm and select “Assign Owner.”

4. In the Assign Owner window, select the required owner from the drop-down list. See Figure 6.

5. Click Assign to commit the change.

Note: The owner can also be changed from the Alarm Details window. See Figure 4

on page 7. Select the owner from the drop-down list. Click Update.

Figure 6. Assign Owner window.

Alarm Forwarding

To forward an alarm:

1. Navigate to the Alarms tab in the Application Landing page.

2. Select an alarm from the Alarm Browser window. See Figure 3 on page 6.

3. Click on the Alarm Forwarding icon , or right click on the selected alarm and select “Alarm Forwarding.”

4. In the Alarm Forwarding window. See Figure 7 on page 10. Select the Trap tab to

forward the alarm to a trap listener. Select Email tab to forward the alarm to an email

recipient.

5. Click on Trap/Email Configuration to add trap/email recipient.

6. Select any trap/email recipient from the list, and select Send to forward the selected

alarm.

Show/Hide

Cleared Alarms

The Show or Hide Cleared Alarms functionality is used to display or hide all the cleared alarms. To show or hide cleared alarms:

1. Navigate to the Alarms tab window in the Application landing page.

(10)

Figure 7. Trap tab on the Alarm forwarding window.

Deleting an Alarm

Users cannot delete alarms generated by the network element. All alarms and logs associ-ated with a network element will be deleted when the network element is removed from the IntelliTeam CNMS network. Administrators and users with sufficient privilege can delete an alarm generated by the IntelliTeam CNMS system. Only those alarms whose severity property is set as Cleared can be deleted.

To delete an alarm:

1. Navigate to the Alarms tab in the Application Landing page.

2. Select an alarm from the Alarm Browser window. See Figure 3 on page 6.

3. Click on the Delete icon , or right click on the selected alarm and select Delete.

Exporting Alarm

It is possible to export the alarms to an Excel /CSV /XML /PDF /HTML file to send it to

users that have no direct access to the IntelliTeam CNMS system. Recipients can even use the export to load data into another trouble-ticket database, and the trouble report can be saved with the alarm details. See the Export section.

(11)

Alarm List Filtering

The list of alarms displayed in the Alarm List can be customized. Filters based on certain parameters, values, and conditions can be set. The Alarm Filter Manager window displays the list of filters and their corresponding filter components.

See Viewing Event Logs with Quick Filters for more information about filtering alarms.

Generating a Report

To generate a report:

1. Navigate to Error Management>Alarms>Generate Reports from the main menu. 2. The Alarm Report Generator window opens. See Figure 8.

3. If you need to generate a report for a specific condition, configure the Conditions pane on the Alarm Report Generator window, see Figure 9 on page 12.

4. Click on Download Report to download the report.

5. The Download Alarm Report File window opens.

6. Right click on Download Exported File to proceed, or click Cancel to cancel the

download process.

(12)

Figure 9. Configure filter conditions on the Alarm Report Generator window.

Configuring

Purge Settings

IntelliTeam CNMS can be configured to periodically purge cleared alarms with these features:

• Clearing of alarms—The inactive alarms are cleared from the network elements. • Periodic Archive and Purge schedule—The IntelliTeam CNMS system can purge

cleared alarms older than one week. After one week of operation, cleared alarms older than seven days are purged daily.

• Periodic Archive and Purge—When the number of cleared alarms increases beyond

a specified limit (for example, 50,000 alarms), excess alarms are purged every 30 minutes.

• Manually triggered by the user—Requests for manual purge of the cleared alarms

are also executed by the IntelliTeam CNMS system.

Because logs are periodically archived, the IntelliTeam CNMS system does not archive the cleared alarms when they are purged.

To purge alarms based on age:

1. Navigate to Error Management>Alarms>Purge Settings on the Main menu 2. The Configure Alarm Purge Interval window opens. See Figure 10 on page 13. 3. Configure and select the check box Purge alarms older than.

(13)

Figure 10. Configure Alarm Purge Interval window.

Purge Alarm Now

To purge all cleared alarms:

1. Navigate to Alarms tab window in the Application Landing page. 2. Click on the Purge Now icon , to purge all cleared alarms.

View Archive

IntelliTeam CNMS archives the alarms as they are purged. The archived files are stored as .csv in the /server/ems/archive directory. This feature allows the user to load and view those archived files through the following options.

To view the purged alarm archive:

1. Navigate to Error Management >Alarms >View Archive from the main menu. 2. The Alarm Archive Viewer window opens. See Figure 11 on page 14.

3. If the .csv file is available on the local hard disk:

• Click on the Load Archive from Local Hard Drive icon . • The Load Archive window opens. See Figure 12 on page 14. • Browse for the archived file.

• Click Load to load the file, or Cancel to abort the process.

4. If the .csv file is available on the IntelliTeam CNMS server: • Click on the Load Archive from Server icon .

• The Load Archive window opens. See Figure 13 on page 14. • Select the archived file from the list.

(14)

Figure 11. Alarm Archive Viewer window.

Figure 12. Load Archive from Local Hard Drive window.

(15)

Alarm Actions

The IntelliTeam CNMS system generates alarms based on periodic monitoring. It also reports operational issues.

The IntelliTeam CNMS system can be configured to take a predefined action when an alarm is received. The following alarm actions are supported by default:

• Automatic SNMP Trap notification to any northbound interface • Automatic Email notification to any email address

Users can define a list of alarms for notification. A specific action will be taken only when an alarm matching the rule/filter is generated. This ensures users do not receive email/notifications for alarms they are not interested in.

Users can also manually forward the alarms to any network agent, such as an SNMP trap, or send an email notification.

Trap Notification

Alarms can be forwarded selectively as traps to northbound clients of other SNMP man-agement systems. The Alarm SNMP Notification window shows the trap destination and alarm filters for traps ready to send. Users can view the forwarded alarms in the alarm browser.

Adding

Alarm Notification

To add an alarm notification:

1. Navigate to Error Management>Manage>SNMP Notification from the main menu. 2. The Alarm SNMP Notification window opens. See Figure 15 on page 16. The left pane

lists the destination addresses and their ports for notifications to be sent. The right pane lists alarm filters.

3. Click Add. The Add New SNMP Listener window opens. See Figure 14.

4. Specify the IP Address (i.e. 192.163.27.32) and Port (i.e. 161).

5. Click OK to commit the data. The newly created SNMP Listener will be listed in the Alarm SNMP Notification window.

(16)

Viewing Alarm SNMP

Notification

To view an alarm SNMP notification:

1. Navigate to Error Management>Manage >SNMP Notification from the main menu. 2. The Alarm SNMP Notification window opens. See Figure 15.

3. The left pane lists the destination addresses and their ports for notifications to be sent. The right pane lists alarm filters.

Figure 15. Alarm SNMP Notification window.

Editing Alarm SNMP

Notification

To edit an alarm SNMP notification:

1. Navigate to Error Management>Manage>SNMP Notification from the main menu. 2. The Alarm SNMP Notification window opens. See Figure 15.

3. Select the SNMP Listener to be modified.

4. Click on the Edit icon . The Edit SNMP Listener window opens. See Figure 16. 5. Make necessary changes, and click OK.

(17)

Associating Listener

Traps with Alarm

Filters

To associate a listener trap with an alarm filter:

1. Navigate to Error Management>Manage>SNMP Notification from the main menu. 2. The Alarm SNMP Notification window opens. See Figure 15 on page 16.

3. Select Trap Destination on the left pane.

4. To select the filters that need to be associated with an alarm, select the required check boxes.

5. Click OK to associate the SNMP listener with alarm filters.

Deleting Alarm SNMP

Notification

To delete an alarm SNMP notification:

3. Select the SNMP Listener to be deleted. 4. Click on the Delete icon .

deletion process.

Refreshing Alarm

SNMP Notification

The Refresh menu icon in the Alarm SNMP Notification window is used to revert to the previous state of the alarm trap forwarder. For example, if an association is selected to perform some operation, and has not yet been saved, the new settings are lost when the refresh button is clicked. To refresh an alarm SNMP notification:

3. Select a trap listener and associate it with any filter. 4. Click on the Refresh icon .

5. Click OK in the confirmation window to proceed, or click Cancel to abort the refresh

process.

Configuring an Alarm

Email Notification

The email server can be configured for an alarm arrival that requires generation of an email alert. The IntelliTeam CNMS server can either send all the alarms or just those that match predefined criteria. To send a notification for specific alarms, create an alarm filter with the match criteria and use the filter to configure the email notification. For example, a filter with the match criteria of owner=“user ID,” would forward all the alarms assigned to that user to his/her email address.

The following alarm details will be sent as part of the email sent to the user: • Alarm ID

• Network element

• Source (shelf/slot/card/port) • Severity

(18)

• Owner assigned

• Acknowledgement status

• Acknowledged time (if acknowledged) • Alarm description

• Remedy (if available)

If the mail server has not already been configured: 1. A pop-up confirmation window is displayed.

2. Click Yes to proceed with the configuration. The Mail Server Configuration window

opens. See S&C Instruction Sheet 1073-530 Mail Server Settings section.

Adding Alarm

Email Recipient

To add an alarm email recipient:

1. Navigate to Error Management>Manage>Email Notification from the main menu. 2. The Alarm Email Notification window opens. See Figure 18 on page 19.

3. Click on the Add icon . The Add New Email Listener window opens. See Figure 17.

4. Specify the Email address and Details. 5. Click OK to commit the data.

(19)

Viewing Alarm

Email Recipient

To view the alarm email recipient:

1. Navigate to Error Management>Manage>Email Notification from the main menu. 2. The Alarm Email Notification window opens. See Figure 18. The left pane lists the

email recipient, who will receive the email. The right pane lists the alarm filters. 3. Select the required email recipient. The filters that have a check mark in the

Associate check box are associated with this email recipient.

Figure 18. Alarm Email Notification window.

Editing Alarm

Email Recipient

To edit the alarm email recipient:

1. Navigate to Error Management>Manage>Email Notification from the main menu. 2. The Alarm Email Notification window opens. See Figure 18.

3. Select the email recipient to be modified.

4. Click on the Edit icon . The Edit Email Recipient window opens. See Figure 19. 5. Make any necessary changes, and click OK.

(20)

Associating Email

Recipient with an

Alarm Filter

To associate the email recipient with an alarm filter:

3. Select an email recipient.

4. To select the filters to be associated with the email recipient, select the required check boxes.

5. Click OK to associate an email recipient with the alarm filters.

Deleting Alarm

Email Recipient

To delete an alarm email recipient:

3. Select the email recipient to be deleted. 4. Click on the Delete icon .

deletion process.

Refreshing Alarm

Email Recipient

To refresh an alarm email recipient:

3. Select an email recipient and associate it with any filter.

4. Click on the Refresh icon . A pop-up Confirmation window opens. 6. Click OK to proceed, or click Cancel to abort the refresh process.

Logs

All errors are logged in the error log with a timestamp. This is carried out irrespective

of their source (alarm logs, event logs, and IntelliTeam CNMS-generated logs). All logs contain the following information:

• Log ID (Unique across all the devices)

• Event Severity—Informational for events and specific severity for alarms

generated by the network element and CNMS

• Network Element

• Log Source—The source could be any component within the network element • Failure Message—A message detailing the reason for failure

• Log Time—The timestamp of the error

• Log Category—For example: fan failure or boot failure

(21)

Alarm Control Settings

The Alarm Control Panel allows administrators to enable/disable individual alarm types and to automatically assign them to specific users. See Figure 20.

1. Navigate to Applications Settings>Alarm Control Settings from the Start menu. 2. Uncheck boxes to disable an individual alarm type.

3. Select a user from the drop-down menu in the Owner column to automatically assign an alarm to a specific user.

4. Click Save to commit the changes, or click Cancel to disregard the changes.

(22)

Event Logs

When an event is generated as the consequence of an error, by default an event log is generated by the CNMS server that is then consolidated into an alarm. These event logs can be viewed, printed, exported and purged.

Figure 21. Events window.

The IntelliTeam CNMS server has a very flexible log browser that allows browsing through all the logs displayed on multiple pages. By default, the log displays the complete set for all the members of a global group. The log browser can also display logs for a selected network element.

The display can be:

• Organized as pages when there are a large number of log entries • Sorted based on one or more columns

• Restricted by selecting the columns to be displayed (or hiding some of the columns) • Filtered based on conditions, so only logs of interest are displayed

To view event logs:

1. Select a group from the hierarchy tree.

2. Navigate to the Events tab on the Application Landing page.

(23)

Figure 22. Events-View window.

Log Details

To view details of an event:

1. Select a network element from the hierarchy tree.

2. Navigate to the Event Logs tab on the Application Landing page.

3. The EventsLog window opens, and displays the list of events along with the details. See Figure 21 on page 22.

4. Select an event log in the list.

5. Click on the View Event Logs icon , or right click and select the View menu. The

Events-View window is displayed. See Figure 22.

6. Click Close to close the Events-View window.

Printing Event Logs

Details displayed on the window can be printed on a standard printer connected to the client machine.

Exporting Logs

This option enables exporting the logs to an Excel/CSV/XML/PDF/HTML file to make it easy to send it to users who have no direct access to the IntelliTeam CNMS system.

Event Log Filters

This feature enables you to view the event logs selectively, based on a user-defined filter condition. Filter management functionality is comprehensive and guided by the intuitive user interface.

See the Filter Management section on page 13 of S&C Instruction Sheet 1073-530.

Viewing Event Logs

with Quick Filters

(24)

Figure 23. Configure Log Purge Interval window.

Purging Logs

The IntelliTeam CNMS system has an archive and purge mechanism with a configurable

Purge Interval for purging older logs. Logs are archived before being purged from the

database. The following purging options are available:

• Periodic archive and purge schedule with a configurable interval. For example, the IntelliTeam CNMS system can be set to purge event logs more than 1 week old. After 1 week of operation, logs older than 7 days will be purged daily.

• Periodic archive and purge if the number of logs increases beyond a specified limit, such as 50,000 logs.

• Manual archive and purge of logs older than the configured number of days upon user request.

• Manual archive and purge of logs more than a certain number of logs. • Manual purge of all the logs with or without archiving the purged logs. To configure the purge interval for event logs:

1. Navigate to Error Management>Events>Purge Settings from the main menu. The

Configure Log Purge Interval window opens. See Figure 23.

2. Configure Purge logs older than and Maximum no. of logs to retain as required. 3. Click OK to save the settings.

Purge Now

To purge all the event logs now:

1. Navigate to Error Management>Events>Purge Now from the main menu. 2. A confirmation message box is displayed.

(25)

View Archive

The IntelliTeam CNMS system archives the events when they are purged. Archived files are stored as a .csv file in the /server/ems/archive directory. To load and view those archived files:

1. Navigate to Error Management >Events >View Archive from the main menu. 2. The Log Archive Viewer window opens. See Figure 24.

3. If the .csv file is available on the local hard disk:

a. Click on the Load Archive from Local Hard Drive icon . b. The Load Archive window opens. See Figure 25 on page 25. c. Browse for the archived file.

d. Click Load to load the file, or Cancel to abort the process.

4. If the .csv file is available on the IntelliTeam CNMS Server: a. Click on the Load Archive from Server icon .

b. The Load Archive window opens. See Figure 26 on page 25. c. Select the archived file from the list.

d. Click Load to load the file, or Cancel to abort the process.

(26)

Figure 25. Load Archive from Local Hard Drive window.

Figure 26. Load Archive from Server window.

Error Stats

The Error Stats window gives a pictorial representation of the alarms and events that have occurred over a period of time. To view the Alarm Stats or Event Stats window. 1. Navigate to the Error Stats tab. See Figure 27.

2. Select Alarm Stats or Event Stats in the drop-down menu.

3. Click on the Edit Dashboard icon to change any chart parameters. 4. Click on the New Dashboard icon to create and save a new dashboard.

(27)

(28)

Operation Error Management