Chapter 3—Data Ingestion
Use of Control Data
After installing the OFSBDF software, you can use control data provided to test end-to-end processing of data (that is, running data ingestion, executing scenarios, and viewing generated alerts in the Alert Management UI). Thus, you can verify that installation of the software is correct and works as designed.
To prepare the system for testing, follow these steps:
1. Complete the prerequisites for using control data (refer to section Prerequisites for Using Control Data on page 89 for more information).
2. Prepare for ingestion of the control data (refer to section Control Data Ingestion on page 89 for more information).
3. Install the control data (refer to section Loading Control Data Thresholds on page 90 for more information).
4. Run Behavior Detection on control data to generate alerts (refer to section Running Behavior Detection on Control Data on page 90 for more information).
Prerequisites for Using Control Data
Before you use control data to test your Behavior Detection installation, the following prerequisites must be fulfilled:
1. The maximum lookback that control data considers is of 13 months, which is for change in behavior scenarios.
Hence, while creating control data ensure that it is spread over 25 different dates in 13 months.
2. The current day according to control data is 20091210.
3. Unless specified, set the current date as 20091210, to generate alerts on control data, before running Behavior Detection Framework.
Note: For more information about control data on your site, contact your OFSBDF Administrator.
Control Data Ingestion
Control data uses a specific set of dates to ensure that all the OFSBDF lock-stock scenarios are tested using this data. The maximum lookback that control data considers is of 13 months, which is for change in behavior scenarios.
The control data is spread over 25 different dates in 13 months. The dates (YYYYMMDD format) being used by control data are:
Table 38. Dates used by Control Data
20081231 20091123
20090130 20091124
20090227 20091125
20090331 20091126
20090430 20091127
20090529 20091130
Use of Control Data Chapter 3—Data Ingestion
On all these dates, run the complete Behavior Detection batch and ingest the data for the respective date. Except for Behavior Detection and Post-Processing tasks, perform all other activities for the Control Data ingestion dates.
Activities required during any Behavior Detection Framework business day are - START BATCH > DRM > DATA INGESTION > BEHAVIOR DETECTION > POST PROCESSING > END BATCH.
Prior to running Behavior Detection on the control data, you must complete the following procedures.
1. Copy all control data from the golden data directory in the database subsystem (/database/golden_data directory) to the Ingestion Manager /inbox directory (refer to section inbox Subdirectory on page 76 for more information).
2. Run ingestion for all the control data ingestion dates. Refer to section Process Flow, on page 34, for more information about the ingestion process.
Note: You need to adjust the partitions of the database tables as per the new dates, if you intend to process Control Data after the database upgrade to OFSBDF 6.2.1.
Loading Control Data Thresholds
To generate breaks on the control data, specific threshold sets and jobs are created. These threshold sets must be installed to the Behavior Detection system for use of control data and generation of test alerts.
1. Navigate to the directory <OFSBDF Installed Directory>/database/golden_data/threshold_sets. This directory consists of test threshold sets of all the scenarios that are available with the OFSAAI system.
2. Execute shell script load_tshld_set.sh. This shell script installs the control data threshold sets for all the scenarios that are installed at your site. It also creates new jobs and template group ID’s corresponding to all the scenarios installed. These template group ID’s are same as the scenario ID’s of installed scenarios.
3. Once the control data thresholds are installed, the system is ready for a test run, that is, generating test alerts.
Running Behavior Detection on Control Data
In order to generate alerts on the ingested control data, execute the new scenario jobs. These jobs consists of same template group ID as the scenario ID. (Refer to Chapter 5, Behavior Detection Jobs, on page 169 to get information regarding about running Behavior Detection Jobs.)
20090630 20091203
20090731 20091204
20090831 20091208
20090930 20091209
20091030 20091210
20091201 20091202
20091121
Table 38. Dates used by Control Data
Use of Control Data Chapter 3—Data Ingestion
Important Notes
1. Run loaded scenarios with the system date as 20091210 with the following exceptions:
a. For Portfolio Pumping scenario, the system date must be 20091204 b. For Active Trading scenario, the system date must be 20091130
2. Check for system errors in the appropriate logs (refer to Appendix A, Logging, on page 309, for more information).
3. Run post-processing procedures.
4. Close the batch to enable display of alerts in the Behavior Detection UI.
5. Log in to the Behavior Detection UI with the correct user credentials.
6. Verify that you can view alerts in the UI.
The display of alerts signifies that installation of the system is correct and works as designed.
Note: The alerts that you can view depend on your user privileges.
Resetting the Environment
Once the testing is complete, you can reset the environment to its clean state by purging the test alerts (refer to section Alert and Case Purge Utility, on page 221, for more on purging the alerts) and deleting the control data threshold sets. The delete_thresholds.sh shell script helps you to delete control data thresholds from the environment.
The delete_thresholds.sh script:
is present in the <OFSBDF Installed Directory>/database/db_tools/bin folder.
takes SCNRO_ID (mandatory) and TSHLD_SET_ID (optional) as parameters.
deletes the threshold sets that are NOT in the default Behavior Detection Product range of sequence IDs (113000000 to 117000000).
Use of Control Data Chapter 3—Data Ingestion