BMC Knowledge Base Development Reference Guide.pdf

(1)

BMC Knowledge Base

Development Reference Guide

Supporting

BMC Event and Impact Management 2.0

BMC ProactiveNet Performance Manager 8.0

(2)

Contacting BMC Software

You can access the BMC Software website at http://www.bmc.com. From this website, you can obtain information about the company, its products, corporate offices, special events, and career opportunities.

United States and Canada

Address BMC SOFTWARE INC

2101 CITYWEST BLVD HOUSTON TX 77042-2827 USA Telephone 713 918 8800 or 800 841 2031 Fax 713 918 8000

Outside United States and Canada

Telephone (01) 713 918 8800 Fax (01) 713 918 8000

BMC, BMC Software, and the BMC Software logo are the exclusive properties of BMC Software, Inc., are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. All other BMC trademarks, service marks, and logos may be registered or pending registration in the U.S. or in other countries. All other trademarks or registered trademarks are the property of their respective owners.

AIX and IBM are registered trademarks of International Business Machines Corporation in the United States, other countries, or both. ITIL® is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office, and is used here by BMC Software, Inc., under license from and with the permission of OGC. Linux is the registered trademark of Linus Torvalds.

Oracle is a registered trademark of Oracle Corporation.

Sun, Java, and Solaris are trademarks or registered trademarks of Sun Microsystems, Inc., in the U.S. and several other countries. UNIX is the registered trademark of The Open Group in the US and other countries.

The information included in this documentation is the proprietary and confidential information of BMC Software, Inc., its affiliates, or licensors. Your use of this information is subject to the terms and conditions of the applicable End User License agreement for the product and to the proprietary and restricted rights notices included in the product documentation.

Restricted rights legend

U.S. Government Restricted Rights to Computer Software. UNPUBLISHED -- RIGHTS RESERVED UNDER THE COPYRIGHT LAWS OF THE UNITED STATES. Use, duplication, or disclosure of any data and computer software by the U.S. Government is subject to

restrictions, as applicable, set forth in FAR Section 52.227-14, DFARS 252.227-7013, DFARS 252.227-7014, DFARS 252.227-7015, and DFARS 252.227-7025, as amended from time to time. Contractor/Manufacturer is BMC SOFTWARE INC, 2101 CITYWEST BLVD, HOUSTON TX 77042-2827, USA. Any contract notices should be sent to this address.

(3)

Customer support

You can obtain technical support by using the BMC Software Customer Support website or by contacting Customer Support by telephone or e-mail. To expedite your inquiry, see “Before contacting BMC.”

Support website

You can obtain technical support from BMC 24 hours a day, 7 days a week at http://www.bmc.com/support. From this website, you can

■ read overviews about support services and programs that BMC offers

■ _{find the most current information about BMC products}

■ _{search a database for issues similar to yours and possible solutions} ■ _{order or download product documentation}

■ _{download products and maintenance} ■ _{report an issue or ask a question}

■ subscribe to receive proactive e-mail alerts when new product notices are released

■ _{find worldwide BMC support center locations and contact information, including e-mail addresses, fax numbers, and}

telephone numbers

Support by telephone or e-mail

In the United States and Canada, if you need technical support and do not have access to the web, call 800 537 1813 or send an e-mail message to [email protected]. (In the subject line, enter SupID:<yourSupportContractID>, such as SupID:12345). Outside the United States and Canada, contact your local support center for assistance.

Before contacting BMC

Have the following information available so that Customer Support can begin working on your issue immediately:

■ _{product information}

— product name

— product version (release number)

— license number and password (trial or permanent)

■ _{operating system and environment information}

— machine type

— operating system type, version, and service pack or other maintenance level such as PUT or PTF — system hardware configuration

— serial numbers

— related software (database, application, and communication) including type, version, and service pack or maintenance level

■ sequence of events leading to the issue

■ _{commands and options that you used}

■ _{messages received (and the time and date that you received them)}

— product error messages

— messages from the operating system, such as file system full — messages from related software

(4)

(5)

Figures

Output from mgetinfo kbsources argument . . . 29

Event processing rule phases . . . 35

Enumeration definition syntax . . . 51

Class definition example . . . 56

Class hierarchy definition example . . . 56

Superclass definition example . . . 56

Subclass definition example . . . 57

Data class definition example . . . 57

Interface class definition example . . . 57

CORE_EVENT class hierarchy . . . 62

CORE_DATA class hierarchy . . . 72

Permitted integer combinations in rules . . . 82

Rule syntax . . . 228

Event selection criteria example . . . 230

when condition triggered by any change to a specified slot . . . 237

when condition triggered by a specific change to a specified slot . . . 238

Rule containing a when clause . . . 239

Sample data . . . 239

Execute rule using dynamic data . . . 243

Interface instance example . . . 246

Refine rule processing . . . 259

Refine rule syntax . . . . 260

Refine rule ECF syntax . . . 260

Refine rule example . . . 261

Filter rule processing . . . 263

Filter rule syntax . . . 263

Event condition formula in a filter rule . . . 264

Filter rule example . . . 265

Regulate rule syntax form 1 . . . 267

Regulate rule syntax Form 2 . . . 267

Regulate rule syntax to send a custom event . . . 268

Regulate rule example 1 . . . 269

regulate rule example 2 . . . 269

Regulate rule example 3 . . . 269

(12)

Correlate rule syntax . . . 280

Correlate rule example 1 . . . 281

Correlate rule example 2 . . . 282

Execute rule syntax . . . 283

When clause in an Execute rule . . . 284

Execute rule example 1 . . . 285

Execute rule example 2 . . . 286

Threshold rule processing . . . 287

Threshold rule syntax . . . 288

Threshold rule example . . . 288

Propagate rule example . . . 291

Timer rule syntax . . . . 292

Timer rule example 1 . . . 293

Timer rule example 2 . . . 293

Delete rule syntax . . . . 294

Delete rule example . . . 294

Collector definition syntax . . . 298

Collector tree definition example . . . 302

Static collector example 1 . . . 303

Static collector example 2 . . . 304

Self collector definition . . . 306

Catchall collector definition . . . 306

MC_SMC_EVENTS collector definition . . . 309

Policy class syntax . . . . 312

Policy entry syntax . . . 312

Policy in a rule syntax . . . 313

Selector class syntax . . . 313

(13)

Tables

Knowledge Base subdirectories . . . 22

Knowledge Base file extensions and directories . . . 23

Cell rule phases . . . 35

BAROC syntax symbols . . . 46

Core and metaclass event and data classes . . . 49

Slot facets . . . 51

MC_EVENT_CATEGORY event categories . . . 53

Default Knowledge Base class files . . . 60

CORE_EVENT base class slots . . . 63

MC_CELL_CONTROL slot definitions . . . 69

PPM_EV slot definitions . . . 70

ABNORMALITY slot definitions . . . 70

ALARM slot definitions . . . 71

CORE_DATA slot definitions . . . 73

BEM_MATCH_TABLE class attribute (slot) definitions . . . 74

POLICY slot definitions . . . 77

SELECTOR slot definitions . . . 77

MC_CELL_INFO slot definitions . . . 78

Deprecated slot substitution . . . 79

Data types . . . 82

Logical combination operators . . . 83

==/2 arguments . . . . 85 !=/2 arguments . . . 86 </2 arguments . . . 86 <=/2 arguments . . . . 87 >/2 arguments . . . 87 >=/2 arguments . . . . 88 between/2 arguments . . . 88 within/2 arguments . . . 89 outside/2 arguments . . . 89 contains/2 arguments . . . 90 contained_in/2 arguments . . . 91 contains_one_of/2 arguments . . . 91 has_prefix/2 arguments . . . 92 has_suffix/2 arguments . . . 92 matches/2 arguments . . . 93 ip_smaller_or_equals/2 arguments . . . 94

(14)

superclass_of/2 arguments . . . 97

subclass_of/2 arguments . . . 98

Alphabetical list of primitives and functions . . . 100

action_requestor/1 syntax argument . . . 107

action_requestor/2 syntax arguments . . . 108

action_requestor/3 syntax arguments . . . 109

action_return/2 syntax arguments . . . 110

admin_execute/5 syntax arguments . . . 111

admin_execute/5 syntax arguments . . . 112

perform/3 arguments . . . 113 perform/5 arguments . . . 113 execute/4 arguments . . . 114 confirm_external/2 arguments . . . 116 get_external/4 arguments . . . 117 inttostring/2 arguments . . . 118 int_to_hex/2 arguments . . . 119 int_to_hex/3 arguments . . . 119 realtostring/2 arguments . . . 120 pointertostring/2arguments . . . 120 string/2 arguments . . . 121 stringtoint/2 arguments . . . 121 stringtoreal/2 arguments . . . 122 stringtopointer/2 arguments . . . 122 int/2 arguments . . . . 123 trunc/2 arguments . . . 123 round/2 arguments . . . 124

real/2 or float/2 arguments . . . 125

code/2 arguments . . . 125 char/2 arguments . . . 126 max/3 arguments . . . 126 min/3 arguments . . . 127 sign/2 arguments . . . 127 abs/2 arguments . . . 128 sqrt/2 arguments . . . 128 exp/2 arguments . . . 129 pow/3 arguments . . . 129 log/2 arguments . . . 130 log10/2 arguments . . . 130 sin/2 arguments . . . . 131 cos/2 arguments . . . 131 tan/2 arguments . . . 132 asin/2 arguments . . . 132 acos/2 arguments . . . 133 atan/2 arguments . . . 133 atan2/3 arguments . . . 134 gcd/3 arguments . . . 134 random/3 arguments . . . 135 incr/1 arguments . . . 135

(15)

incr/2 arguments . . . 137 incr/3 arguments . . . 137 incr/3 arguments . . . 138 incr/4 arguments . . . 138 decr/1 arguments . . . 139 decr/2 arguments . . . 140 decr2 arguments . . . 140 decr/3 arguments . . . 141 decr/3 arguments . . . 142 decr/4 arguments . . . 142 concat/2 arguments . . . 143

strlen/2 and len/2 arguments . . . 143

tolowercase/2 and lower/2 arguments . . . 144

touppercase/2 and upper/2 arguments . . . 145

strpart/3 arguments . . . 146 strnpart/4 arguments . . . 146 strextract/4arguments . . . 147 substring/3 arguments . . . 148 substring/4 arguments . . . 148 strip/2 arguments . . . 149 strip/3 arguments . . . 149

Possible values for the $POS argument . . . 150

strip/4 arguments . . . 150

Possible values for the $POS_{argument . . . 151}

strtolist/3 arguments . . . 152 strmatch/3 arguments . . . 152 match_regex/3 arguments . . . 154 match_regex/4 arguments . . . 155 match_regex/5 arguments . . . 157 sprintf/3 arguments . . . 158 mapslots/4 arguments . . . 158 mapslots/3 arguments . . . 159

has_substring/2 syntax argument . . . 160

has_substring/3 syntax argument . . . 161

time_stamp/1 arguments . . . 162 time_stamp_to_cim/2 arguments . . . 162 time_stamp_to_str/3 arguments . . . 163 time_stamp_to_str/2 arguments . . . 164 time_extract/3 arguments . . . 164 str_to_time_stamp/3 arguments . . . 166 listlen/2 arguments . . . 167 listgetelt/3 arguments . . . 167 listmember/2 arguments . . . 168 listdelete/3 arguments . . . 169 listappend/3 arguments . . . 169 listdisjoint/2 arguments . . . 170

(16)

listremdup/2 arguments . . . 172 listwalk/2 arguments . . . 173 add_to_list/2 arguments . . . 173 rem_from_list/2 arguments . . . 174 find_match/5 arguments . . . 174 find_match_entry/4 arguments . . . 177 apply_match_entry/4 arguments . . . 178 get_list_slotvalues/3 arguments . . . 179 set_list_slotvalues/3 arguments . . . 180 class_path/2 arguments . . . 181 reset_default/1 arguments . . . 182 ntadd/2 arguments . . . 182 ntcnt/2 arguments . . . 182 ntget/5 arguments . . . 183 ntset/3 arguments . . . 184 opadd/4 arguments . . . 184 opadd/3 arguments . . . 185 opcnt/2 arguments . . . 185 opget/7 arguments . . . 186 opget/6 arguments . . . 187 opget_time/3 arguments . . . 187 opget_author/3 arguments . . . 188 opget_action/3 arguments . . . 188 opget_args/3 arguments . . . 189 opset/5 arguments . . . 190 opset/4 arguments . . . 190 relate/1 arguments . . . 191 unrelate/1 arguments . . . 192 cellinfo/2 arguments . . . 193 cellcontrol/1 arguments . . . 194 kbversion/2 arguments . . . 195 kbversion/1 arguments . . . 196 get_env/2 arguments . . . 196 send_to/2 arguments . . . 197 send_to/3 arguments . . . 197 send_to_ext/4 arguments . . . 198 propagated_to/3 arguments . . . 199 smcomps/5 arguments . . . 200 key_version/2 arguments . . . 202 key_verify/2 arguments . . . 203 key_verify/3 arguments . . . 203 tf_active/1 arguments . . . 204 tf_active/2 arguments . . . 205 tf_udid_active/1 arguments . . . 206 tf_udid_active/2 arguments . . . 206 tf_current_start/2 arguments . . . 207 tf_current_start/3 arguments . . . 207 tf_current_end/2 arguments . . . 208

(17)

tf_current_interval/2 arguments . . . 209 tf_current_interval/3 arguments . . . 209 tf_prev_start/2 arguments . . . 210 tf_prev_start/3 arguments . . . 211 tf_prev_end/2 arguments . . . 211 tf_prev_end/3 arguments . . . 212 tf_prev_interval/2 arguments . . . 212 tf_prev_interval/3 arguments . . . 213 tf_next_start/2 arguments . . . 214 tf_prev_start/3 arguments . . . 214 tf_next_end/2 arguments . . . 215 tf_next_end/3 arguments . . . 215 tf_next_interval/2 arguments . . . 216 tf_next_interval/3 arguments . . . 216 tf_duration/3 arguments . . . 217 tf_duration/4 arguments . . . 217 generate_event/2 arguments . . . 218 new_data/3 arguments . . . 219 remove_data/1 arguments . . . 220 set_timer/3 arguments . . . 222 set_timer_at/3 arguments . . . 223 set_timer_at/4 arguments . . . 223

Syntax object description . . . 228

Conditions for the using clause . . . 234

MC_CELL_PARSE_ERROR slots . . . 254

MC_CELL_UNDEFINED_CLASS slots . . . 254

MC_CELL_PROCESS_ERROR slots . . . 255

Filter rule syntax descriptions . . . 264

f1 and f2 Filter rules event processing examples . . . 265

Correlate rule event examples . . . 282

Available environment variables in Execute rules . . . 284

BMC Impact Manager standard roles . . . 301

By Status collector set . . . 307

By Location collector set . . . 307

Collectors included in the MCxP collector set . . . 308

Property groupings: BMC_BaseEvent class . . . 319

Data types . . . 319

CEM to BAROC: Metadata . . . 320

CEM to BAROC: source information . . . 320

CEM to BAROC: reporter information . . . 321

CEM to BAROC: situation information . . . 321

CEM to BAROC: metric information . . . 321

EventInformation::EventToCIAssociationType parameter values . . . 323

ReportTime (optional) . . . 324

EventModelVersion (required) . . . 325

(18)

Notes (optional) . . . 326 EventToCIAssociationType (optional) . . . 326 PropagationHistory (required) . . . 327 RelationSource (required) . . . 327 Owner (optional) . . . 327 Account (optional) . . . 328 ResourceId (optional) . . . 328 ReconciliationIdentity (recommended) . . . 328 Alias (recommended) . . . 329 ComponentHost (required) . . . 329 ComponentHostAddress (required) . . . 329 Location (optional) . . . 329 ComponentURI (optional) . . . 330 ComponentCaption (recommended) . . . 330 ComponentType (recommended) . . . 330 ComponentOwner (optional) . . . 331

Slots for event monitoring information . . . 331

ResourceId (optional) . . . 332 ComponentHostAddress (required) . . . 332 ComponentURI (optional) . . . 332 ComponentCaption (required) . . . 333 ComponentType (optional) . . . 333 EventTime (required) . . . 333 EventType (optional) . . . 333 EventId (required) . . . 334 EventSeverity (optional) . . . 334 EventSuggestion (optional) . . . 334 SituationCategory (required) . . . 335

Situation category (mc_event_category) values . . . 335

SituationTime (required) . . . 337 Priority (optional) . . . 337 Severity (required) . . . . 338 Message (recommended) . . . 338 Application (optional) . . . 338 LongMessage (optional) . . . 339 RepeatCount (optional) . . . 339 MetricName (optional) . . . 339 MetricValue (optional) . . . 339 MetricValueUnit (optional) . . . 340 MetricThreshold (optional) . . . 340

(19)

C h a p t e r

1

Working with the Knowledge Base

This chapter presents the following topics:

What is a Knowledge Base? . . . 20

Components of a Knowledge Base . . . 20

How a KB is created. . . 20

Knowledge Base index files . . . 24

Managing a Knowledge Base . . . 24

Integrating a unified KB with pre-7.2 cell definitions . . . 25

Creating a new production or test Knowledge Base—mcrtcell . . . 25

Importing Knowledge Base information into a cell—mkb . . . 25

Compiling a Knowledge Base—mccomp . . . 26

Loading a Knowledge Base into a running cell—mcontrol . . . 26

Implementing changes to a Knowledge Base . . . 27

Versioning a Knowledge Base . . . 27

Event classification and formatting . . . 30

How event classes are structured . . . 30

How the Knowledge Base classifies incoming events . . . 32

Event processing . . . 33

Rules . . . 33

Event management policies . . . 33

How an event management policy differs from a rule . . . 33

When to use an event management policy rather than a rule . . . 34

How events are processed using rules . . . 34

Event collectors. . . 41

MetaCollectors (only available for BMC Impact Solutions) . . . 43

(20)

What is a Knowledge Base?

A Knowledge Base (KB) defines the behavior of a cell. KB classes define what

information is contained in each event. KB rules define how the events are processed. You can modify the KB to customize its behavior in your environment.

The KB is similar to a script and the cell is the engine that runs the script. The KB is a compiled collection of files, such as event processing rules, class definitions, and executables, organized in a directory structure. The KB files are loaded by a cell at start time. The Knowledge Base instructs the cell how to format incoming event data, process received events, and display events in a console. Although many KBs can exist within a distributed environment, each cell can be associated with only one KB at a time.

How a KB is created

During installation, a KB that serves as a template for all cell KBs is created for the cell. This KB provides the cell with the data definitions, data instances, collector definitions, and rules for a fully functional environment in which to process events and service components. If you modify the template KB, any cell that you install or create will include those modifications.

When you create or install a new cell using the mcrtcell command, you always create or install a KB in the newly-created cell’s KB directory path:

MCELL_HOME/etc/CellName/kb. Modifications to the KB in the CellName/kb directory apply to the CellName cell only.

Components of a Knowledge Base

A KB includes the following:

■ event classes define the types of events to accept and classify source event data for

processing

■ data classes define the classes and slots of dynamic data instances and service model

component instances.

■ dynamic data function as contextual variables that can provide data values to rules

(21)

Knowledge Base directory structure

■ global records are persistent structured global variables that maintain data values

across all phases of event processing.

■ event management rules are event processing statements that use the BAROC data

associated with an event, data instances or records to determine if, when, and how to respond to new events or event modifications.

■ event management policies are one of several generic rule types that perform actions

against events that meet selection criteria specified in an associated event selector. An event management policy selects the events that you want to process, defines the processes needed to manage those events, and schedules when the events are processed.

■ event collectors are filters that query the event repository and display the results in

an event list in an organized manner.

■ action executables are executable programs or scripts that perform an automated

task on a particular event.

Knowledge Base directory structure

The Knowledge Base uses a defined directory structure to organize its files and executables. The Knowledge Base directories are in the following locations:

■ The Knowledge Base used by the cell during runtime is located in

%MCELL_HOME%\etc\CellName\kb on Windows platforms and in $MCELL_HOME/etc/CellName/kb on UNIX platforms.

■ The template Knowledge Base resides in the %_{MCELL_HOME%\etc\default}

\standard or $MCELL_HOME/etc/default/standarddirectory.

Cells are created during installation of a cell or by using the mcrtcell command. For information about this command, see the Administration Guide.

NOTE

The environment variables created during installation that define paths to cell configuration files and executables are listed in the Installation Guide.

(22)

Figure 1 lists the directory structure for a Knowledge Base.

In the Knowledge Base, each subdirectory is labeled to indicate the type of files or programs it stores, as listed in Table 1 on page 22.

Figure 1 Knowledge Base directory structure

kb \bin \A \h1 \l2 \p4 \s5 \w4 \classes \collectors \data \lib \records \rules

Table 1 Knowledge Base subdirectories (part 1 of 2)

Knowledge Base

subdirectory Description

bin stores the external scripts that can execute during rule processing and actions

The bin directory organizes the scripts and programs in subdirectories specific to the appropriate operating system, as follows:

■ A—independent, all UNIX, or non-Windows ■ h1—HP-UX

■ l2 —Linux ■ p4 —AIX ■ s5 —Solaris ■ w4 —Windows

The .load file in the bin directory specifies the order in which external scripts or programs are presented to clients. Actions are defined in .mrl files. There is one default file, .load, in the bin directory.

classes stores event class, data class, and interface definitions

Classes are stored in .baroc files. The .load file in the classes directory specifies the order in which classes are loaded. Parent classes must be loaded prior to child classes.

(23)

Table 2 lists the file extensions and directory location for the each of the components contained in a KB.

collectors stores collector rule definitions

Collector definitions are used to organize the event lists that are viewed in the console. Collector rules are defined in .mrl files. Collectors and their syntax are described in Chapter 7, “Working with collectors.”

data instances of dynamic data stored in files that are loaded when the cell is initialized

Dynamic data instances are stored in .baroc files. The .load file indicates the order in which the files are loaded into the cell. After the values are loaded into the cell any changes are

maintained in the mcell.db. Dynamic data objects and their syntax are described in Chapter 5, “Event rules.”

lib stores primitives and functions used in the Knowledge Base

For example, the Knowledge Base contains the following files that cannot be modified:

■ _sim.wic_{—contains the compiled implementation of primitives and functions that are}

loaded by the cell at startup

■ _{sim_decl.wic}_{—contains the compiled definitions for primitives and functions; it is loaded}

by the compiler to compile rules that reference SIM primitives

For more information about functions and primitives, see Chapter 4, “Master Rule Language (MRL) reference.”

records stores global record definitions, which store dynamic information across all rule phases A global record stores persistent dynamic information in a .baroc file. Many rule processing phases use global records for retrieving dynamic information. The .load file indicates the order in which the files are loaded into the cell. The default copy of record definitions is stored in baroc files in the records directory. After the values are loaded they are maintained in the mcell.db. Dynamic data objects and their syntax are described in Chapter 5, “Event rules.” rules stores the rule definitions for the Knowledge Base

The source for rule definitions are the files with an .mrl extension. The compiled versions of rules are contained in files with the .wic or .pkg extension. The .load file indicates the order in which the rules are loaded into the cell. Rules and their syntax are described in Chapter 5, “Event rules.”

Table 2 Knowledge Base file extensions and directories (part 1 of 2)

Component File extension Directory

Table 1 Knowledge Base subdirectories (part 2 of 2)

Knowledge Base

(24)

Knowledge Base index files

The following files are included with the installation and are necessary for the Knowledge Base to run properly:

■ _manifest.kb—serves as an index file for the listed directories that compose the

Knowledge Base during compilation. This file is located in

%MCELL_HOME%\etc\CellName\kb on Windows platforms and in $MCELL_HOME/etc/CellName/kb on UNIX platforms.

■ _.load—serves as an index file for the individual files contained in the

corresponding subdirectory of the Knowledge Base directory structure. Load files are included in each subdirectory to determine load order for that particular directory. Files types within the .load file do not have extensions.

■ ._loadwic—Before the compilation of the Knowledge Base, rules and collectors are

created in .mrl files and are included in the .load files. After compilation, rule and collector files are stored in .wic files and a .loadwic file is created for the KB to use. The .wic files are machine-readable only.

Managing a Knowledge Base

To manage a Knowledge Base, you must perform several tasks by using the cell command-line interface (CLI). This section briefly describes these tasks; for more information about syntax and options available for the CLI commands, see the Administration Guide.

global records .baroc kb\records

rules .mrl kb\rules

collectors .mrl kb\collector

action executables .mrl kb\bin

service model class definitions .baroc kb\classes

interface classes .baroc kb\classes

scripts and programs not applicable kb\bin\platform

Table 2 Knowledge Base file extensions and directories (part 2 of 2)

(25)

Integrating a unified KB with pre-7.2 cell definitions

In version 7.2.00, the unified Knowledge Base was introduced. You can integrate your cell definitions from cells older than version 7.2.00 with the unified KB of the current version of the cell.

1

Create a new cell using the mcrtcell CLI with either the -ae or -as option.

2

Copy the modifications or extensions you’ve made in old cell’s KB to the new cell’s KB.

To do so, you can manually edit the files or use your specific utilities.

3

Recompile the KB, and restart the cell.

Creating a new production or test Knowledge Base—mcrtcell

Use the mcrtcell command to create a new production or test cell and Knowledge Base. The mcrtcell command can be run only on the local computer where the cell is being created. For more information about syntax and options available with mcrtcell, see the Administration Guide.

Importing Knowledge Base information into a cell—mkb

You can use the mkb command to import an existing Knowledge Base. You can also use this command to import files containing definitions for event classes, interfaces, global records, data classes, collectors, or rules from an existing Knowledge Base. For more information about syntax and options available with mkb, see the

NOTE

To protect the format of the default Knowledge Base, back it up prior to making any

modifications. An adequate backup includes all directories and files in the kb directory or the directory where the changes occur.

You can also use source-control programs such as CVS or Subversion to keep track of changes to the KB. Source control allows you to revert to older versions of the KB and to examine changes.

(26)

Compiling a Knowledge Base—mccomp

Each time you change, add, or delete actions, classes, collectors, or rules, you must compile the KB. The cell recognizes changes to the KB only when the cell is restarted. Use the mccomp command to compile the Knowledge Base. The mccomp command parses event, data class and global records, and compiles the rules. For more information about syntax and options available with mccomp, see the Administration Guide.

Effects of compiling a Knowledge Base with tracing

enabled

If you enable tracing by using the -t option when compiling a KB and if

TraceRuleToXact=Yes in mcell.conf, an event can be tracked in the transaction log, an

.xact file, as it progresses through the rule execution. Entries in the log file related to rule tracing are include a TRCX header.

However, deploying a KB compiled with the -t option can degrade performance by as much as 50 percent. BMC recommends that you do not use the -t option to compile the production KB.

Loading a Knowledge Base into a running cell—mcontrol

You must load a KB on a running cell each time that you edit collectors. Use the mcontrol reload kb command to reload the Knowledge Base while the cell is still running. For more information about the mcontrol command, see the Administration Guide.

NOTE

To use the mkb command to manipulate an existing KB, you must use the -f parameter to define the path to the manifest.kb file and specify the action that the mkb command should execute.

NOTE

The TraceRuleLevel parameter in the mcell.conf file must be set to 2 for rules tracing to occur.

(27)

Implementing changes to a Knowledge Base

You must stop and start the cell to implement any changes to a cell’s KB. For instructions on stopping and starting a cell, see the Administration Guide.

Versioning a Knowledge Base

KB versioning enables you to determine which KB and which version of the KB is loaded in a cell. You can implement version information for

■ _{KB source files}— For each KB source file that you specify, information about the

source file is provided and the version of the compiler that was used to compile it.

■ _{Logical KB modules}—Version information is provided for each logical module that

you identify in the KB.

A logical KB module is a collection of class definitions and rules that perform a specific task within the KB. For instance, all class definitions and rules that are related to Help Desk events could be called the HelpDesk KB module. A single KB can contain multiple such logical modules. The class definitions and rules that are not associated to a specific KB module are considered to be part of the global, unnamed KB module.

If desired, you can make rules behave differently depending on the version of specific KB modules. This can be useful in patches, for example.

Enabling KB versioning

To enable versioning, you must create logical modules in the KB. To identify the files for a particular module, add the @kbversion annotation to the KB source files, using the following syntax:

@kbversion( [ ModuleName , ] VersionID )

Variable Description

ModuleName specifies the name of the module to which the current file belongs

To indicate version information for the global module, either use the empty string as ModuleName or omit ModuleName.

(28)

Versioning a Knowledge Base

The mccomp command compiles the @kbversion annotations into the KB object files and includes the following information about each source file in the KB:

■ release number of the compiler used to compile the file ■ build number of the compiler used to compile the file ■ build date of the compiler used to compile the file ■ source file name

■ source file size in bytes ■ source file checksum

KB versioning example

This example specifies that the KB contains a logical module called HelpDesk, and that its version is 1.2.01.

Retrieving KB version information in rules

You can retrieve KB module version information in a rule by using the kbversion primitive. For information about the kbversion primitive, see “kbversion/1 — retrieve global Knowledge Base module version information” on page 196.

Retrieving KB version information by using a command—

mgetinfo

You use the mgetinfo command with the kbmodules and kbsources arguments to retrieve version information from the cell's loaded KB.

WARNING

Multiple @kbversion annotations for the same module will result in a compilation error. This also applies to a global version; only one annotation without a module name is allowed in a KB.

@kbversion( HelpDesk , '1.2.01' )

mgetinfo -n cellName [-v] kbmodules|kbsources

Argument Returned results

kbmodules list of KB modules with version information

(29)

Versioning a Knowledge Base

The information is displayed in raw format. You can use the -v switch to obtain the information in a more readable format. Figure 2 shows a portion of the information returned from the kbsources argument.

In addition, the KB also includes a reference copy of

■ the BMC Atrium CMDB (Configuration Management Database) Common Data

Model (CDM) Service class definitions used in a cell’s service model

■ the service model for the cell, published by a BMC Impact Publishing Server

Figure 2 Output from mgetinfo kbsources argument

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 268 3028595382 collectors/self_collector.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 848 2351849679 collectors/pom_activeevents_collectors.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 1298 1572060265 collectors/pom_intelligentevents_collectors.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 2276 3736979305 collectors/catchall_collector.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 882 4217293348 collectors/pom_byuser_collectors.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 5411 3623989645 collectors/ibrsd_collectors.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 143 4015122127 rules/kbversem.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 145 3230031018 rules/kbverssim.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 455 3870777253 rules/mc_startup.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 1348 4183429197 rules/refine_multiple_server_events.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 35175 1041057786 rules/im_internal.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 3009 1294038250 rules/mc_intevt.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 2364 3554485946 rules/impact_admin_server.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 2595 287947525 rules/ips.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 1896 51731806 rules/mc_sm_start.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 3885 2912043582 rules/mc_sm_associate.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 11587 1724466729 rules/mc_ci_policies.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 2126 3683660578 rules/mc_sm_maintenance.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 957 4072896080 rules/mc_sm_elect.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 1865 2174473777 rules/mc_sm_attach.mrl

Compiler: 8.x.xx (Build: xxx - 5-Nov-2xxx) Source file (size/checksum/name): 3885 3336615242 rules/mc_sm_shadow.mrl

(30)

Event classification and formatting

The event source (an event adapter, an integration application, another cell, an API, or a CLI) must provide events in the BAROC format and structure. The cell accepts and processes an incoming event if it matches an event class definition in the Knowledge Base (KB).

The following sections explain how event classes are structured and how the Knowledge Base uses event classes to classify and format incoming events.

How event classes are structured

An event class defines the types of events that the cell will accept and classifies source event data for processing. For example, one class of event might be Microsoft

Windows operating system events. That event class has several subclasses: application events, security events, and system events. Each event subclass represents a common type of event that occurs, such as a network logon, that contains varying data values, such as a time stamp and the name of the host on which the logon occurred. The varying data values from a source event are stored in data fields called slots (attributes). An actual event is an instance of an event class.

Using the BAROC language, you define each event class and its slots. Each slot has a data type and can have specific attributes, called facets, that can control the values that the slot can have or control aspects of the event’s processing. Slot enumerations specify acceptable values for a particular slot.

You can think of a class definition as an empty form with its input fields representing slots, and a class instance as a completed form. A valid instance must respect slot types, and if some slot values are not specified, they are implicitly set to their default values, which are inherited from the parent class.

The following example illustrates the event class structure for Windows Event Log classes: Class: CORE_EVENT Class: EVENT Class: MC_ADAPTER_BASE Class: WIN_EVENTLOG Class: WIN_EL_APPLICATION Class: WIN_EL_SECURITY Class: WIN_EL_SYSTEM

(31)

How event classes are structured

The following shows the same example with some of the slots defined for these classes:

In this example, a WIN_EL_APPLICATION event defines mc_tool as Application. Because WIN_EL_APPLICATION is a sub-class of WIN_EVENTLOG, it inherits the mc_tool_class slot definition of WINEventLog. WIN_EL_APPLICATION also inherits mc_client_address from the CORE_EVENT class. mc_client_address contains the network address of the host of the adapter that sent the event.

By comparison, a WIN_EL_SECURITY event defines mc_tool as Security; however, it inherits the same values for mc_tool_class and mc_client_address as a

WIN_EL_APPLICATION event.

Event classes and their syntax are described in Chapter 3, “Event and data classes.”

Class inheritance

BAROC class definitions are organized in a hierarchical system where existing classes (superclasses) can be assigned subclasses so that the subclasses automatically inherit definitions from these superclasses. This behavior is called inheritance.

While a subclass inherits all the slot definitions of the superclass, it can also contain additional new slot definitions of its own, and even slot definitions that override a superclass slot definition. However, when a subclass slot overrides a superclass slot definition, it cannot have a different data type from the inherited slot, only different facet values.

Also, a rule defined for a class applies to all instances of its subclasses. For example, a rule defined for the base event class, EVENT, applies to all events because all event classes are subclasses of EVENT.

Class: CORE_EVENT - Flags: p

Slot: mc_client_address - Type: STRING - Flags: rkpdh - Def: Class: EVENT

Class: MC_ADAPTER_BASE Class: WIN_EVENTLOG

Slot: mc_tool_class - Type: STRING - Flags: rkPDh - Def: WINEventLog Class: WIN_EL_APPLICATION

Slot: mc_tool - Type: STRING - Flags: rkPdh - Def: Application Class: WIN_EL_SECURITY

Slot: mc_tool - Type: STRING - Flags: rkPdh - Def: Security Class: WIN_EL_SYSTEM

(32)

How the Knowledge Base classifies incoming events

In summary, subclasses

■ inherit slots from superclasses ■ inherit rules from superclasses ■ can have their own slots

■ can override superclass slots (but must contain the same data type) ■ can be a subclass of only one class

How the Knowledge Base classifies incoming events

When the event is collected by the cell, the cell compares the incoming event to the predefined event classes in the Knowledge Base to determine the event type of the incoming event and validate the event.

If an incoming event does not match one the predefined event classes, the cell takes one of the following actions:

■ If the event does not match any of the predefined events in the KB at all, an internal

event in the form of an error message is generated and the event is stored as an MC_CELL_UNDEFINED_CLASS event with a MINOR severity and a slot, class_name, containing the original, incorrect event class.

■ If the event is of a class that matches one of the predefined event classes, but

contains undefined event slot(s), the event is generated and continues to processing, but incorrect slot name(s) are stored in the mc_bad_slot_names slot and the corresponding value(s) are stored in the mc_bad_slot_values slot.

■ If the event has slot(s) that contain value(s) that cannot be interpreted (for example,

alphabetical string data in an integer slot), the default slot value(s) are used, the incorrect slot name(s) are stored in the mc_bad_slot_names slot, and the value(s) of the incorrect slot(s) are stored in the mc_bad_slot_values slot.

■ If an event cannot be parsed, an internal MC_CELL_PARSE_ERROR event is

created containing the text for that event stored in the event_text slot. The MC_CELL_PARSE_ERROR event also uses error_line, error_column and

error_messages slots to indicate the position in text where the error occurs and the parsing error message. The MC_CELL_PARSE_ERROR event has a default

severity of MINOR.

Interface classes

Interface classes are used to interface with external programs. They define the format of data that comes from the external source, allowing the external information to be

(33)

Event processing

Once events are collected and formatted, they are processed by the cell rules engine. You can control how incoming events are processed using either rules or event management policies.

Rules

Rules are processing statements that determine and control the behavior of cells. A rule determines if and how events are processed. Rules consist of a set of statements that evaluate whether or not an event is processed. If the event is to be processed the rule can include an event management function or action to perform, such as

discarding the event, enriching the event data, automatically escalating an event, or automatically executing an action on the event. You write rules using the Master Rule Language (MRL). Rules are compiled and stored in the cell’s KB. Rules and rule syntax are described in Chapter 5, “Event rules.”

Event management policies

An event management policy is one of several generic rule types that perform actions against events that meet selection criteria specified in an associated event selector. An event management policy selects the events that you want to process, defines the processes needed to manage those events, and schedules when the events are processed.

Event management policies can be defined interactively using predefined, out-of-the-box policy types accessed through the console. You can also create new user-defined policy types to add new event processing actions.

Whether predefined or user-defined, all event management policies consist of

■ event selector ■ process(es) ■ timeframe(s) ■ evaluation order

(34)

When to use an event management policy rather than a rule

However, unlike rules, an event management policy is easily defined interactively through the console rather than being manually written in MRL uses an event selector by which you specify the criteria used to select events for processing by the policy. The event selector allows you to specify a number of events that meet selection criteria. This gives the event policy greater flexibility than a rule. does not require compilation because it is implemented using predefined data classes and precompiled rules.

When to use an event management policy rather than a rule

Use a policy if there is a fairly simple, routine action that you would like to apply to many events. If some complex event manipulation is required that is specific to a small subset of events, a rule written in MRL may be more appropriate.

In some cases, a rule can provide better performance than its event management policy equivalent. If an event management policy gives problematic performance, substituting an equivalent rule might rectify the performance issue.

How events are processed using rules

Rules are associated with a specific rule phase based on their type; each phase represents a logical stage of event processing. The cell processes each incoming event one phase at a time and evaluates each event against one rule at a time. Internal events are always processed before external events. The order in which the cell evaluates events against rules is determined by the order in which the rules were loaded.

Figure 3 on page 35 identifies the rule phases and shows how event processing proceeds, and Table 3 on page 35 describes the phases.

(35)

How events are processed using rules

Figure 3 Event processing rule phases

Table 3 Cell rule phases (part 1 of 2)

# Rule phase Description

1 Refine validates incoming events and, if necessary, collects additional data needed before the event is processed further

2 Filter identifies events that should be discarded

2 Regulate evaluates events, and, if evaluated as true, collects duplicate events for a time period. If a specified threshold of duplicates is reached, the Regulate phase passes an event to the next processing phase.

4 New determines which events in the event repository should be updated with new information from new incoming events

During this phase:

■ _{actions are triggered that must be performed just before a new event comes in}

Propagate Filter Regulate

Abstract The New, Abstract,

Correlate, and Execute rule phases can trigger a Timer rule.

Delete rules execute when a record is removed during repository cleanup. 8 9 5 3 2 1 _Refine Event Delete 11 Timer 10 Repository New 4 An event can be discarded during any one of these phases before being added to the repository. Threshold 8 8 Execute 7 Correlate 6 to other cells

(36)

Rule phase processing overview

The cell uses a main loop that starts transactions for each triggering event, such as:

■ a new incoming event

■ a timer expiration (by timer rule unless it is part of a regulate rule) ■ end of a process called by the get_external or confirm_external primitive ■ cleanup

■ modification of a slot by a client (such as the console, or the mposter or msetmsg

CLIs) or the propagation of this modification from another cell

When each of these transactions occur, all applicable rules are executed in the order of the phases (as shown in Figure 3 on page 35), then in order of definition inside each phase.

Starting at the New phase:

■ every slot modification to the current event or other existing events are logged in a

first-in, first-out queue

■ abstraction events created by the abstraction rules are stored in another queue ■ internal events generated by the generate_event() primitive are put in another

first-in, first-out queue

5 Abstract evaluates events, and, if certain conditions are met, triggers the generation of abstraction events.

An abstraction event is a summary event based on other events that are occurring. 6 Correlate determines whether any events have a cause-and-effect relationship

7 Execute specifies actions to perform when a slot of a new event matches a condition, or a slot of an old event is modified to satisfy a condition

8 Threshold specifies the actions that must be performed when a certain number of duplicate events have been received over a certain time period

9 Propagate determines whether an event is forwarded to another cell or integration product 10 Timer specifies actions to be executed when a timer has expired.

A timer can be set in the New, Abstract, Correlate, Execute, Threshold and Delete phases. 11 Delete triggers actions to ensure that data integrity is maintained when an event is deleted from the

event repository during the cleanup process

Table 3 Cell rule phases (part 2 of 2)

(37)

When all the rules have been evaluated the queue slot modifications are processed (triggering the when parts of the rules). During this processing, slot modifications and internal events continue to be added to the queues.

When the slot modification queue is empty, the abstraction events are processed starting directly at the new phase. When all abstraction events have been processed, the internal event queue is processed.

When all internal events have been processed, the processing continues with the next transaction in the main loop.

Rule phase processing details

Each incoming event's processing begins with the Refine phase. A Refine phase rule allows the rule engine to start an external program to confirm the event or to obtain more information from the environment. The processing of this event is suspended until the external program returns its results. The rule engine is not idle because it processes other events in the meantime.

After the Refine phase, the incoming event enters the Filter phase where it can be discarded if it does not meet the requirements for processing. If the event is discarded in the Filter phase, it is not stored in the event repository, so it cannot be accessed by any future processes that rely on the event repository.

The Regulate rule phase, like the Refine phase, can suspend the evaluation process while it waits for events similar to those held to appear before a time window closes. Depending on the rules and circumstances, the event is then discarded or it continues the evaluation process through the remaining rules.

The New phase is the last in which the rule engine can discard an incoming event without its entering the event repository. After this phase, an event can be dropped only if it is explicitly deleted or discarded during the cleanup process.

The event then goes through the final phases, Abstract, Correlate, Execute, Threshold, and Propagate, in that order, if any rules have been defined for those phases.

Abstract rules can be used to create high-level, or abstract, events based on low-level events, such as a SERVERS_LOGIN_ATTACK event based on certain

LOGIN_FAILURE events.

NOTE

The Refine phase and the Regulate phase are the only phases in which the evaluation process may be suspended. In all other phases, the event is processed sequentially through all the rules of all phases.

(38)

Correlate rules can be used to build an effect-to-cause relationship between an event that occurs as a result of another event, such as relating certain APP_DOWN events to certain APP_MISSING_PROCESSES events.

Execute rules can be used to perform specified actions when a slot value has changed in the repository.

Threshold rules can be used to count the number of events that matches the criteria you specify; if the number of these events exceeds the amount allowed within a time frame the Threshold rule executes.

Propagate rules forward events to other cells; such as, escalation of an event from a lower-level cell to a higher-level cell.

Internal event processing by rules

Internal events are processed in the same way as external events, but have a higher priority than external events.

The rule engine processes all the internal events before accepting new external events. All events are process in first-in, first-out order. For example, if, during the evaluation of an internal event, another internal event is generated, the rule engine processes the second internal event first.

Internal requests by a rule

Internal requests are actions that a rule requests during the processing of an event. The rule engine processes all internal requests in a first-in, first-out order before anything else is processed. Examples of internal requests are

■ modify the contents of a slot in an event ■ monitor the returning remote actions

■ perform cleanup by removing old event data ■ apply a time window for a Regulate rule

Maintaining rules

You can use dynamic data and global records to make it easier to maintain and manage rules.

(39)

Dynamic data

Dynamic data is similar to a small database within the cell. Dynamic data function as contextual variables that can provide data values to rules and policies during event processing. By using dynamic data, you can create generic event management rules and policies that apply broadly. This greatly simplifies the creation and maintenance of the event management rules.

For example, without using dynamic data, if you want to change the severity of an event based on the host name of a device, you must create a rule for each host name. Using dynamic data, you can define the host names and corresponding severity as data instances and reference them from one generic rule, rather than writing one rule for each possible host name.

To define a dynamic data instance, you must first define a new data class. You can use the Dynamic Data Editor to define data class instances for use in event management rules or service models. As new hosts are added to the environment, you can add new data instances dynamically through the using the CLI or an API, or through event management rules themselves. You do not need to recompile event

management rules to use new data instances.

Dynamic data is stored in the event repository and updated whenever the context changes while the cell is running.

For more information about dynamic data, data classes, and the Dynamic Data Editor, see the Administration Guide.

Global records

Global records are persistent, structured global variables that maintain data values across all phases of event processing. Their scope is the entire Knowledge Base; any other type of variable has a scope limited to the current rule. Global records are addressed by name. You can use global records to share information between events during event processing.

(40)

For example, this record

is used in a rule in im_internal.mrl to set a default value to the location slot of the following event:

It also is used in the following rule:

For more information about global records, see “Global record definition syntax” on page 58.

Using event management policies for event processing

There are two types of event management policies—predefined event management policies and user-defined event management policies. Predefined event management policies are provided out-of-the box. User-defined policies are custom policies that you create.

Each event management policy defines selection criteria that is applied to incoming events to determine which events are processed. A timeframe determines when the policy is active or inactive. The evaluation order determines which policies are implemented first if there is a conflict.

RECORD EM_KB_OPTIONS DEFINES {

startup_script_enabled: MC_YESNO, default = NO; default_location: STRING; } END ... if ($EM_KB_OPTIONS.default_location != "") then { $EV.mc_location = $EM_KB_OPTIONS.default_location; } ... refine exec_script_at_cell_startup: MC_CELL_START ($START) where

[

$EM_KB_OPTIONS.startup_script_enabled == YES ]

{

execute($START, "startup_script", [], 'yes'); }

(41)

Event collectors

Predefined event management policies

The two types of predefined event management policies are standard and dynamic data enrichment.

A standard event management policy requires you to use the console to input data into a policy. This type of policy works well if you only want to apply the policy to a small number of events or hosts.

A dynamic data enrichment policy provides additional context to an event by extracting data from an external source and appending it to the event so it is accessible to IT operations. For example, it may be useful for you to know the location of a particular piece of equipment. This type of information is not normally included in a standard technical event; however, you can use dynamic data enrichment to add this

information to the event by accessing data stored external to the product (for example, an asset store). If you want to apply a policy to a large number of hosts or events, you should use a dynamic data enrichment policy.

Dynamic data enrichment policies require the same components as standard event management policies. However, dynamic enrichment policies allow you to import external enrichment data into the policy, rather than having to enter it manually. An external enrichment data source can provide additional information about an event that is not available from the technology from which the event originates. An example of an external enrichment data source is a database such as an asset data store.

User-defined event policies

Predefined policies cannot cover all requirements of different product

implementations. To support specialized event processing, you can also define and implement custom event policy types to do specialized event processing not supported by the predefined policy types. For instructions on creating event policy types, see the Administration Guide.

Event collectors

You can use event collectors to organize events in meaningful groups for display in an event list and to show event relationships. An event collector consists of a filter that queries the event repository and displays the results in an event list in the console. Each cell has default collectors for the console and collectors that you create. In the

BMC Knowledge Base Development Reference Guide.pdf