IBM SmartCloud Analytics - Log Analysis
Anomaly App
Version 1.2
IBM SmartCloud Analytics - Log Analysis
Anomaly App
Version 1.2
Note
Before using this information and the product it supports, read the information in “Notices,” on page 7.
Edition notice
This edition applies to IBM SmartCloud Analytics - Log Analysis and to all subsequent releases and modifications until otherwise indicated in new editions.
References in content to IBM products, software, programs, services or associated technologies do not imply that they will be available in all countries in which IBM operates. Content, including any plans contained in content, may change at any time at IBM's sole discretion, based on market opportunities or other factors, and is not intended to be a commitment to future content, including product or feature availability, in any way. Statements regarding IBM's future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only. Please refer to the developerWorks terms of use for more information.
© Copyright IBM Corporation 2014.
Contents
Chapter 1. About this publication . . . . 1
Audience . . . 1
Publications . . . 1
Accessing terminology online . . . 1
Accessibility . . . 1
Tivoli technical training. . . 1
Providing feedback . . . 2
Conventions used in this publication . . . 2
Typeface conventions . . . 2
Chapter 2. Anomaly app . . . 3
Configuring the Anomaly app training . . . 3
Running model training . . . 4
Appendix. Notices . . . 7
Chapter 1. About this publication
This guide contains information about how to use IBM SmartCloud Analytics - Log Analysis.
Audience
This publication is for users of the IBM SmartCloud Analytics - Log Analysis product.
Publications
This section provides information about the IBM SmartCloud Analytics - Log Analysis publications. It describes how to access and order publications.
Accessing terminology online
The IBM Terminology Web site consolidates the terminology from IBM product libraries in one convenient location. You can access the Terminology Web site at the following Web address:
http://www.ibm.com/software/globalization/terminology.
Accessibility
Accessibility features help users with a physical disability, such as restricted mobility or limited vision, to use software products successfully. In this release, the IBM SmartCloud Analytics - Log Analysis user interface does not meet all
accessibility requirements.
Accessibility features
This information center, and its related publications, are accessibility-enabled. To meet this requirement the user documentation in this information center is provided in HTML and PDF format and descriptive text is provided for all documentation images.
Related accessibility information
You can view the publications for IBM SmartCloud Analytics - Log Analysis in Adobe Portable Document Format (PDF) using the Adobe Reader.
IBM and accessibility
For more information about the commitment that IBM®has to accessibility, see the IBM Human Ability and Accessibility Center. The IBM Human Ability and
Accessibility Center is at the following web address: http://www.ibm.com/able (opens in a new browser window or tab)
Tivoli technical training
Providing feedback
We appreciate your comments and ask you to submit your feedback to the IBM SmartCloud Analytics - Log Analysis community.
Conventions used in this publication
This publication uses several conventions for special terms and actions, operating system-dependent commands and paths, and margin graphics.
Typeface conventions
This publication uses the following typeface conventions: Bold
v Lowercase commands and mixed case commands that are otherwise difficult to distinguish from surrounding text
v Interface controls (check boxes, push buttons, radio buttons, spin buttons, fields, folders, icons, list boxes, items inside list boxes,
multicolumn lists, containers, menu choices, menu names, tabs, property sheets), labels (such as Tip:, and Operating system considerations:) v Keywords and parameters in text
Italic
v Citations (examples: titles of publications, diskettes, and CDs v Words defined in text (example: a nonswitched line is called a
point-to-point line)
v Emphasis of words and letters (words as words example: "Use the word
that to introduce a restrictive clause."; letters as letters example: "The
LUN address must start with the letter L.")
v New terms in text (except in a definition list): a view is a frame in a workspace that contains data.
v Variables and values you must provide: ... where myname represents.... Monospace
v Examples and code examples
v File names, programming keywords, and other elements that are difficult to distinguish from surrounding text
v Message text and prompts addressed to the user v Text that the user must type
v Values for arguments or command options
Chapter 2. Anomaly app
Configure the anomaly app so that IBM SmartCloud Analytics - Log Analysis users can use it to help them to identify and investigate anomalies in the log data. Before IBM SmartCloud Analytics - Log Analysis users can use the app, you must configure the Anomaly app for your installation of IBM SmartCloud Analytics -Log Analysis
After you configure the app, you must contact a data scientist. The data scientist runs the data model training. When the data scientist runs the training, the app creates a baseline of historical data. This baseline is modeled as a histogram that is based on the data that is ingested by IBM SmartCloud Analytics - Log Analysis during the time period that is required by the model. The app supports two statistical models for data analysis, the mean-variance model and the auto-regression model.
After the app is configured and the model training is complete, IBM SmartCloud Analytics - Log Analysis users can use the app to identify anomalous events. The app uses historical data to detect deviations from normal system behavior. This behavior is derived from time-stamped text data from logs or events that are indexed in IBM SmartCloud Analytics - Log Analysis. It does not use structured performance measurement data.
To open the app, log in to IBM SmartCloud Analytics - Log Analysis. After the configuration and model training are complete, the app is displayed under Custom apps.
The app displays a histogram for each data source. The app calculates the
anomalies for each data source type by comparing current data to the baseline that the data scientist created. If all the data sources for a specified data source type contain anomalous events, the model name is displayed in red. If all the trained models for a data source declare that the current data to be anomalous, the model name is displayed in yellow. If all the trained models for a data source declare that the current is not anomalous, the name is displayed in green.
Configuring the Anomaly app training
Before you can use the Anomaly app, you must configure it so that it works with your installation of IBM SmartCloud Analytics - Log Analysis.
Before you begin
v Extract the anomalyAppTraining.tar file from the <HOME>/AppFramework/CLIApps/ directory to a suitable directory, for example <AnomalyAppTraining>.
v To use this app, you must install R and a required Java add-on, rJava version 0.95. R is a software environment for statistical computing. To install R and the required add-on, complete the following steps:
1. Create a folder that serves as the installation folder, referred to here as <R>. 2. Download the following packages to the new folder:
– rJava_0.9-5.tar.gz from http://www.rforge.net/src/contrib/rJava_0.9-5.tar.gz
3. Copy the installR.sh script that you extracted to the <AnomalyAppTraining> folder in the first prerequisite to the <R> directory. The installR.sh file is located in the <AnomalyAppTraining>/bin directory.
4. To add the location of your IBM SmartCloud Analytics - Log Analysis installation to the installR.sh script, edit the UNITY_HOME parameter.
5. Run the installR.sh command from the <R> directory. The command creates a folder that is called <R>/R-3.0.2 referred to here as <R_HOME>.
R requires specific prerequisite packages. For more information, see:
v http://cran.r-project.org/doc/manuals/R-admin.html#Simple-compilation v
http://cran.r-project.org/doc/manuals/R-admin.html#Essential-and-useful-other-programs-under-a-Unix_002dalike
The installR.sh script assumes that the prerequisites for R, except for the X11 and Redline packages, are already installed. If these prerequisite packages are not installed, an error message is displayed during the installation that lists the missing packages. If this situation arises, install the missing packages, delete the
<R_HOME>directory on your installation, and start the installation again.
Procedure
Open the env.sh script in the <AnomalyAppTraining>/bin directory and add values for the following variables:
<UNITY_HOME>
Add the path to the directory where IBM SmartCloud Analytics - Log Analysis is installed. For example, /home/user/IBM/LogAnalyticsWorkgroup.
<R_HOME>
Add the path for the <R_HOME> directory that you created as a prerequisite.
Results
After you configure the training, a data scientist must run the model training.
Running model training
After you configure the Anomaly app, a data scientist runs the model training.
Before you begin
Before the data scientist can run the model training, the IBM SmartCloud Analytics - Log Analysis administrator or user must configure the training. See “Configuring the Anomaly app training” on page 3.
To complete the procedure, the reader needs both the domain and statistical knowledge that is provided by a specialized data scientist. The procedure is only for use by data scientists.
Procedure
1. In the AnomalyAppTraining/config/trainingFiles folder, create one folder for each source type that you want to use with the app. Use the name of the
source type as the name of the folder. For example, the AnomalyAppTraining/ config/trainingFiles/WASSystemOutfolder is created by default for the WASSystemOutsource type.
2. In the source type folder, create one or more training files for the corresponding source type. Use the sample files that are stored in the AnomalyAppTraining/ config/trainingFilesfolder as a reference.
Specify the following values in the training file: Filter expression
Use the training file to define a filter expression that is applied to the log data stream. The training file changes the log data stream in to an event stream. For example:
severity=E, Node=MyMachine
Time window
Specify a time window during which events from the log data stream are counted for the data source.
Model to be used
Select either a mean variance or regression analysis-based statistical model.
Model-specific training parameter
Specify the parameters for the chosen statistical model. For example, if you use a mean variance-based model, you must specify a value for the Acceptable_P_Valueparameter that used to create the mean variance model.
3. Run the model training. To run the training file, run the run.sh command that is in the AnomalyAppTraining/bin/ folder. The command contains two optional parameters that you can specify:
- logsources=<Comma_seperated_list_of_data_sources>
Use this parameter to list specific data sources. You use this parameter if you want to run the app on the listed data sources. If you do not enter a value, the app runs on all data sources.
- overwrite= true or false
If you want to train a data source that is already trained, set this parameter to true. The already trained data source is trained again. The default value is false.
Results
The app uses the model training to create a historical base line. When the IBM SmartCloud Analytics - Log Analysis users runs the app from the UI, the app compares the current data against the data that is generated by the model training to identify anomalies.
Appendix. Notices
This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing
IBM Corporation North Castle Drive
Armonk, NY 10504-1785 U.S.A.
For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual Property Department in your country or send inquiries, in writing, to: Intellectual Property Licensing
Legal and Intellectual Property Law IBM Japan, Ltd.
1623-14, Shimotsuruma, Yamato-shi Kanagawa 242-8502 Japan
The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law:
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement might not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.
Any references in this information to non-IBM Web sites are provided for
IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.
Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact:
IBM Corporation 2Z4A/101
11400 Burnet Road
Austin, TX 78758 U.S.A.
Such information may be available, subject to appropriate terms and conditions, including in some cases payment of a fee.
The licensed program described in this document and all licensed material
available for it are provided by IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or any equivalent agreement between us.
Any performance data contained herein was determined in a controlled
environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurement may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment.
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of
performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only.
This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.
If you are viewing this information in softcopy form, the photographs and color illustrations might not be displayed.
Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml.
Adobe, Acrobat, PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, other countries, or both.
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Product Number:
