Monitor Your Key Performance Indicators using WSO2 Business Activity Monitor

(1)

Published on WSO2 Inc (http://wso2.com)

Home > Stories > Monitor Your Key Performance Indicators using WSO2 Business Activity Monitor

Monitor Your Key Performance Indicators using

WSO2 Business Activity Monitor

By kasunw.wso2.com Created 2013-06-26 07:07

Applies To

WSO2 BAM Version 2.0.0 and above

Table of Content

Introduction[1] BAM architecture[2] Use case[3]

KPIs for this use case[4] Collect information[5]

BAM data-agent (Java API)[6] Non Java Data-agent[7] REST API[8]

BAM data receiver[9] Cassandra explorer[10] Data Analysis[11]

Writing a hive script for analyzing captured data[12] Visualization[13]

Introduction

Key performance indicators are essential for your organization to measure the performance or success in terms of progress of its goals. Also It is crucial for you to choose the right KPIs with better understanding of what is important to the organization to measure and improve the business productivity of it. WSO2

Business activity monitor has an extremely flexible framework that allows to model your own key

performance indicators to suit your monitoring requirement. In this article I am going to explain how you can build your own KPI using WSO2 Business Activity Monitor.

BAM architecture

(2)

Data that needs to be monitored goes through the above modules and the data flow is as follows. The system that you are going to monitor have the BAM data agents and these agents will

capture the required data and send it to BAM. In addition to the data agents users can send data to BAM through the REST API.

Data receiver in BAM receive this data and store them in Cassandra data store.

Analyzer engine will run the analytics written in hive language according to the monitoring requirement, the summarize data can be persist into RDBMS data store.

In the presentation layer summarized data will fetch from RDBMS and show it in the dashboard/reports.

Use case

I am going to illustrate the capability of defining KPIs by using an use-case. Let's think You have hosted a news website and You need to monitor the traffic comes to your website and analyze it to understand how you can enhance your website and marketing strategies to increase the profit.

KPIs for this use case

No of unique visitors per day

How many visitors does it take for your website to achieve its goals? Your profit will proportional to the no of visitors to your website. If this shows some growth day by day, which means you are making a progress.

Most popular news category

This category will depends on several factors (time, incidents, gossip, events etc.). Hence

analyzing your traffic and identifying the current in demand news categories are essential for you to promote your contents more in those sources and increase the profits.

Number of peoples from different locations

This will helpful for identifying the regions that have low no of visitors and take some actions to improve the traffic from these areas.

Now I am going to illustrate step by step on how to use BAM to define above KPIs.

(3)

Website traffic information can be captured and published into BAM using following ways. Using Java API

Using client generated by thrift IDL Using REST API

BAM data-agent (Java API)

BAM provides high performance, low latency, load-balancing (between multiple receivers ), non-blocking and multi-threaded API for sending large amount of business events over various transport (TCP,Http) using Apache Thrift. This API has been provided as a Java SDK and you can use it easily for publishing captured data from your Java based system to BAM for analysis. In addition to that these data-agents are compatible with WSO2 CEP that can be used for real time analysis.

Custom data-agent

Following I have written a simple asynchronous data-agent for publishing web traffic information to BAM. This will help you to understand the agent API usage. For more information about writing a data publisher for BAM, you can refer this article [1]

import org.apache.log4j.Logger; import org.wso2.carbon.databridge.agent.thrift.Agent; import org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher; import org.wso2.carbon.databridge.agent.thrift.conf.AgentConfiguration; import org.wso2.carbon.databridge.agent.thrift.exception.AgentException; import org.wso2.carbon.databridge.commons.Event;

public class DataAgent {

private static Logger logger = Logger.getLogger(DataAgent.class);

public static final String ONLINE_NEWS_STATS_DATA_STREAM = "online_news_stats"; public static final String VERSION = "1.0.0";

public static final String DATA =

"Kasun,Colombo,Sri Lanka vs Australia 3rd ODI,Sports,1366433587\n" +

"Amal,Kaluthara,Businessman killed in Expressway accident,Accidents,1366533595\n" + "Kamal,Colombo,Navy intelligence investigating boat escape,Military,1366633695\n" + "Kalum,Mathara,No leadership change - JVP,Politics,1366433520\n" +

"Nuwan,Galle,Marginal improvement at this year?s O/L exam results,Education,1366533987\n"+ "Sampath,Mathara,Marginal improvement at this year?s O/L exam results,Education,1366433987\n"+ "Chamath,Mathara,Attempts to replace Minister Sirisena: UNP,Politics,1366634587\n" +

"Prabath,Colombo,WikiLeaks; LTTE could have threatened Karunanidhi,War,1366535587\n" + "Amila,Kandy,Private bus owners to launch one-day strike,Transport,1366732587\n" +

"Budhdhika,Colombo,ICC introduces new 'No ball' playing condition,Sports,1366731587\n" + "Rangana,Nuwara Eliya,Annual horse racing event in Nuwara Eliya,Sports,1366534787\n" + "Gamini,Colombo,O/L 2012 results released,Education,1366434987\n" +

"Tharindu,Colombo,O/L best results,Education,1366334087\n" + "Janaka,Kaluthara,O/L best results,Education,1366839787\n" +

"Anuranga,Jaffna,Sri Lanka vs Australia 3rd ODI,Sports,1366336737\n" +

"Kasun,Colombo,Marginal improvement at this year?s O/L exam results,Education,1366338788\n"+ "Ranga,Kandy,Private bus owners to launch one-day strike,Transport,1366832727\n" +

"Denis,Kaluthara,ICC introduces new 'No ball' playing condition,Sports,1366534117\n" + "Harsha,Galle,No leadership change - JVP,Politics,1366533333\n";

(4)

AgentConfiguration agentConfiguration = new AgentConfiguration();

System.setProperty("javax.net.ssl.trustStore", "/opt/wso2bam-2.2.0/repository/resources/security/client-truststore.jks"); System.setProperty("javax.net.ssl.trustStorePassword", "wso2carbon");

Agent agent = new Agent(agentConfiguration); //Using Asynchronous data publisher

AsyncDataPublisher asyncDataPublisher = new AsyncDataPublisher("tcp://127.0.0.1:7611", "admin", "admin", agent); String streamDefinition = "{" +

" 'name':'" + ONLINE_NEWS_STATS_DATA_STREAM + "'," + " 'version':'" + VERSION + "'," +

" 'nickName': 'News stats'," +

" 'description': 'Stats of the news web site'," + " 'metaData':[" + " {'name':'publisherIP','type':'STRING'}" + " ]," + " 'payloadData':[" + " {'name':'userName','type':'STRING'}," + " {'name':'region','type':'STRING'}," + " {'name':'news','type':'STRING'}," + " {'name':'tag','type':'STRING'}," + " {'name':'timestamp','type':'LONG'}" + " ]" + "}";

asyncDataPublisher.addStreamDefinition(streamDefinition, ONLINE_NEWS_STATS_DATA_STREAM, VERSION); publishEvents(asyncDataPublisher);

}

private static void publishEvents(AsyncDataPublisher dataPublisher) { String[] dataRow = DATA.split("\n");

for (String row : dataRow) {

String[] data = row.split(",");

Object[] payload = new Object[]{data[0], data[1], data[2], data[3], Long.parseLong(data[4])};

Event event = eventObject(null, new Object[]{"10.100.3.173"}, payload); try {

dataPublisher.publish(ONLINE_NEWS_STATS_DATA_STREAM, VERSION, event); } catch (AgentException e) {

logger.error("Failed to publish event", e); }

} }

private static Event eventObject(Object[] correlationData, Object[] metaData, Object[] payLoadData) { Event event = new Event();

event.setCorrelationData(correlationData); event.setMetaData(metaData); event.setPayloadData(payLoadData); return event; } }

Non Java Data-agent

Thrift IDL can be used to generate thrift clients from different languages to publish data. In this case asynchronous data publishing implementations will be to developed by the developers the way similar to

(5)

how it has been implemented in Java API. Languages are supported by thrift

C++ C# Cocoa D Delphi Erlang Haskell Java OCaml Perl PHP Python Ruby Smalltalk REST API

You can capture the website traffic information from your system and use BAM REST API for publishing those data to BAM via Http transport.

BAM data receiver

Data send from above data agents will receive by the data receiver which use thrift and internal optimization techniques to achieve very high throughput. Those received data will directly write to

Cassandra which is high performance and scalable big data storage. It persists events into a column family with name equal to the stream name. Also BAM data receiver can be used share the events with WSO2 CEP for real time analysis.

Cassandra explorer

It provides graphical user interface for viewing the data in a Cassandra column family. Now let's see the data published from our sample client.

Go to Home ? Manage ? Cassandra Explorer ? Connect to Cluster Type the connection details as below.

(6)

Then go to online_news_stats in the EVENT_KS.

You can view the data by going to the column family.

After successfully publish the data we can do the analysis.

Data Analysis

You can run data analytics on captured data by using the BAM analytics framework which is powered by Apache Hadoop for scaling out data processing operations on a large number of data processing nodes, in order to handle large data volumes. WSO2 BAM provides an easy way to do the map/reduce operation on Hadoop by using Apache Hive that provides a simple way to write query and managing large datasets residing in distributed storage using a SQL-like language called HiveQL.

KPIs

Following I have list down the KPIs that we are going to analyze.

No of unique visitors per day Most popular news category

No of peoples from different regions

Writing a hive script for analyzing captured data

This is the Hive script to analyze the above KPIs.

CREATE EXTERNAL TABLE IF NOT EXISTS OnlineNewsStats (key STRING, userName STRING,region STRING, news STRING, tag STRING, payload_timestamp BIGINT) STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' WITH SERDEPROPERTIES ( "cassandra.host" = "127.0.0.1", "cassandra.port" = "9160","cassandra.ks.name" = "EVENT_KS", "cassandra.ks.username" = "admin","cassandra.ks.password" = "admin", "cassandra.cf.name" = "online_news_stats", "cassandra.columns.mapping" = ":key,payload_userName,payload_region,payload_news,payload_tag,payload_timestamp" ); CREATE EXTERNAL TABLE IF NOT EXISTS UniqueVisitorsPerDay(count INT, day STRING) STORED BY 'org.wso2.carbon.hadoop.hive.jdbc.storage.JDBCStorageHandler' TBLPROPERTIES ( 'wso2.carbon.datasource.name'='WSO2BAM_DATASOURCE', 'hive.jdbc.update.on.duplicate' = 'true', 'hive.jdbc.primary.key.fields' = 'day', 'hive.jdbc.table.create.query' = 'CREATE TABLE UNIQUE_VISITORS_PER_DAY ( count INT, day VARCHAR(100))' ); insert overwrite table UniqueVisitorsPerDay select count ( distinct username), from_unixtime(payload_timestamp,'yyyy-MM-dd') as day from OnlineNewsStats group by from_unixtime(payload_timestamp,'yyyy-MM-dd' ); CREATE EXTERNAL TABLE IF NOT EXISTS PopularNewsCategory(count INT, category STRING) STORED BY 'org.wso2.carbon.hadoop.hive.jdbc.storage.JDBCStorageHandler' TBLPROPERTIES ( 'wso2.carbon.datasource.name'='WSO2BAM_DATASOURCE', 'hive.jdbc.update.on.duplicate' = 'true', 'hive.jdbc.primary.key.fields' = 'category', 'hive.jdbc.table.create.query' = 'CREATE TABLE POPULAR_NEWS_CATEGORY ( count INT, category VARCHAR(100))' ); insert overwrite table PopularNewsCategory select count(tag), tag from OnlineNewsStats group by tag; CREATE EXTERNAL TABLE IF NOT EXISTS PeoplesFromDifferentLocations(count INT, region STRING) STORED BY 'org.wso2.carbon.hadoop.hive.jdbc.storage.JDBCStorageHandler' TBLPROPERTIES ( 'wso2.carbon.datasource.name'='WSO2BAM_DATASOURCE', 'hive.jdbc.update.on.duplicate' = 'true', 'hive.jdbc.primary.key.fields' = 'region', 'hive.jdbc.table.create.query' = 'CREATE TABLE PEOPLES_FROM_DIFFERENT_LOCATIONS ( count INT, region VARCHAR(100))' ); insert overwrite table PeoplesFromDifferentLocations select count(region),region from OnlineNewsStats group by region;

RDBMS data storage can be configured from master-datasources.xml in WSO2BAM/repository/conf/datasources/ directory. You need to change the following section

<datasource> <name>WSO2BAM_DATASOURCE</name> <description>The datasource used for analyzer data</description> <definition type="RDBMS"> <configuration> <url>jdbc:h2:repository/database/samples/BAM_STATS_DB;AUTO_SERVER=TRUE</url> <username>wso2carbon</username> <password>wso2carbon</password> <driverClassName>org.h2.Driver</driverClassName> <maxActive>50</maxActive> <maxWait>60000</maxWait> <testOnBorrow>true</testOnBorrow> <validationQuery>SELECT 1</validationQuery> <validationInterval>30000</validationInterval> </configuration> </definition> </datasource>

From BAM 2.3.0 onwards you can use Cassandra datasource in hive script and it can be configured from master-datasources.xml.

Once you run the above script hive read data from the Cassandra storage and summarize it, then the summarized data will persist into RDBMS storage to visualize via dashboard.

Visualization

After you analyze and summarize data according to your KPIs, you need to visualize your kpis. Since those summarized data in the RDBMS storage you can plug third party visualization tools for the visualization. Here you can find an article about using different reporting frameworks with WSO2 Business Activity Monitor [2] Also you can use WSO2 Jaggery framework[3] or WSO2 BAM gadget generation tool to visualize your KPIs.

Here I have used gadget generation tool for generating and deploying gadgets in dashboard. You can find information on how to use it from here [4] Dashboard

Number of Visitors Per Day Popular News Category Visitors Distribution References http://wso2.org/library/articles/2012/07/creating-custom-agents-publish-events-bamcep [14] http://wso2.org/library/articles/2012/09/using-different-reporting-frameworks-wso2-business-activity-monitor [15] http://jaggeryjs.org/ [16] http://docs.wso2.org/wiki/display/BAM220/Gadget+Generation+Tool [17]

Author

Kasun Weranga Gunathilake, Senior Software Engineer, WSO2 Inc. Hide

setTimeout(function(){var a=document.createElement("script");var b=document.getElementsByTagName("script")[0];a.src=document.location.protocol+"//dnn506yrbagrg.cloudfront.net/pages/scripts/0023/3595.js?"+Math.floor(new Date().getTime()/3600000);a.async=true;a.type="text/javascript";b.parentNode.insertBefore(a,b)}, 1);

div#footer { background-color: transparent; } div#iFooter { background-color: #000000; font-family: Arial; position: relative; } div#iFooter div.cFooterMenu { width: 980px; margin: auto; color: #ffffff; padding: 25px 0px 10px; } div#iFooter div.cFooterMenu ul { list-style: none; margin: 0px; padding: 0px; } div#iFooter div.cFooterMenu ul li { border: 0px; } div#iFooter div.cFooterMenu ul li.cTitle { width: 170px; font-size: 14px; line-height: 18px; font-weight: bold; color: #ffffff; float: left; position: relative; padding: 10px 10px 10px; } div#iFooter div.cFooterMenu ul li.cTitle span { border-bottom: solid 1px #666666; display: block; width: 100%; padding: 0px 0px 10px; color: #ffffff; } div#iFooter div.cFooterMenu ul li a { font-family: Arial; color: #ffffff; text-decoration: none; font-size: 11px; line-height: 15px; } div#iFooter div.cFooterMenu ul li a:hover { color: #f47b20; } div#iFooter div.cFooterMenu ul li ul li { font-size: 12px; font-weight: normal; line-height: 12px; padding: 4px 0px; float: none; position: inherit; } div#iFooter div.cFooterMenu ul li ul { margin: 5px 0px 0px; } div#iFooter div.cFooterMenu ul li ul li.cSubTitle { font-weight: bold; color: #cccccc; padding: 10px 0px 5px; border-bottom: dotted 1px #777777; margin: 0px 0px 5px; } div#iFooter div.cFooterLogo { height: 60px; background-color: #000000; } div#iFooter div.cFooterLogo .cContent { width: 980px; height: 60px; margin: auto; text-align: right; background-image: none; } div#iFooter div.cFooterLogo .cContent a img { float: left; margin: 17px 20px 0px 0px; } div#iFooter div.cFooterLogo .cContent a { line-height: 60px; color: #ffffff; text-decoration: none; font-size: 12px; } div#iFooter div.cFooterLogo .cContent a:hover { color: #f47b20; } div#iFooter div.cCopyRight { width: 980px; height: 30px; margin: auto; font-size: 12px; color: #cccccc; line-height: 60px; } div#iFooter div.cFooterLogo span { float: right; line-height: 60px; color: #cccccc; } div#iFooter div.cFooterLogo ul { list-style: none; margin: 0px; padding: 0px; } div#iFooter div.cFooterLogo ul li { float: left; position: relative; padding: 0px 10px; border: 0px; line-height: 60px; } div#iFooter div.cFooterLogo ul li a { color: #cccccc; text-decoration: none; line-height: 60px; } div#iFooter div.cFooterLogo ul li a:hover { color: #f47b20; } div.cClear { clear: both; } @media screen and (max-width: 900px) { div#iFooter div.cFooterLogo .cContent { width: 100%; } div#iFooter div.cFooterLogo span { width: 100%; margin: auto; text-align: center; height: 20px; } div#iFooter div.cFooterMenu { display: none; } } #iFooter sup { font-size: 10px; font-style: italic; color: #f16021; } WSO2 Advantage Company Customers Partners --> Products Middleware Platform Overview API Manager App Manager Application Server

Business Process Server Business Rules Server Carbon

Complex Event Processor Data Analytics Server Dashboard Server Data Services Server

Enterprise Mobility Manager Enterprise Service Bus Enterprise Store Governance Registry Identity Server IoT Server Machine Learner Message Broker

Microservices Framework for Java New Process Center New Storage Server Development Tools Developer Studio Cloud Cloud Overview Private Cloud

WSO2 Private PaaS WSO2 App Factory Managed Cloud Managed Cloud

Managed Cloud Overview Service Level Agreement --> Public Cloud App Cloud API Cloud Use Cases Technology Challenges IT Challenges Business Challenges --> Events Events Calendar Webinars Workshops WSO2Con Conferences Training Partner Events Resources Library Overview Articles | Tutorials Analyst Reports Case Studies On-Demand Webinars Presentations White Papers Documentation Videos

Support & Training Support

Training

Support System Login Subscribe to the newsletter Legal

Privacy

var _bizo_data_partner_id = "7067";(function() { var s = document.getElementsByTagName("script")[0]; var b = document.createElement("script"); b.type = "text/javascript"; b.async = true; b.src = (window.location.protocol === "https:" ? "https://sjs" : "http://js") + ".bizographics.com/insight.min.js"; s.parentNode.insertBefore(b, s);})();

[4] http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#KPIs for this use case [5] http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#Collect information

[6] http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#BAM data-agent (Java API) [7] http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#Non Java Data-agent [8] http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#REST API

[9] http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#BAM data receiver [10] http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#Cassandra explorer [11] http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#Data Analysis

[12] http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#Writing a hive script for analyzing captured data [13] http://wso2.com/library/articles/2013/06/monitor-your-kpis-using-wso2-bam#Visualization

[14] http://wso2.org/library/articles/2012/07/creating-custom-agents-publish-events-bamcep

[15] http://wso2.org/library/articles/2012/09/using-different-reporting-frameworks-wso2-business-activity-monitor [16] http://jaggeryjs.org/