Session D15
Simple Visualization of your
TimeSeries Data
Shawn Moe IBM
Agenda
• IoT & Gateways
• Moving sensor data around • jQuery and Ajax
• Data Access Options
Acknowledgements and Disclaimers
Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM
operates.
The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.
© Copyright IBM Corporation 2015. All rights reserved.
— U.S. Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
IBM, the IBM logo, ibm.comare trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or TM), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at
•“Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml
Internet of Things Topography
Streams
Deep Analytics Zone
Device/Sensors
Smart Gateways Sensor Analytics Zone
What are we trying to do?
• Develop an IoT Gateway Kit to give various
interested parties a framework to explore IoT Gateway benefits:
– Business Partners
– Hackathon participants – Hobbyists
• Gateway kit will demonstrate:
– How to get sensor data into the Informix database – How to visualize the data
– How to aggregate and establish alert conditions – How to transport the data to the Cloud
Two key challenges for the Gateway
1. Working with sensors & getting data from sensor to the Gateway database
– ~200 different protocols used by sensors
– Established protocols for various industry devices
2. Visualizing the data in the gateway
– Simple current & historical status visualization – Simple analytics acting on information stored on
the gateway and from the cloud – Aggregation where appropriate
Getting data into the Gateway
• Node-RED framework (nodered.org) used to “wire” things together visually
– IBM backed open source project – also very visible on Bluemix
– Light-weight runtime is built on Node.js, taking full advantage of its event-driven, non-blocking model – 120,000 modules in Node's package repository.
Contributions help interactions with various devices, sensors, actions, analytics, and external SW packages – We have developed & contributed timeseries node
Getting data into the Gateway
• Besides Node-RED, other methods we have experimented with:
– Python – JavaScript
• This code is available at our GitHub project: https://github.com/IBM-IoT/ibm-sample-code
Getting data to the Cloud
• MQTT (was Message Queue Telemetry Transport
• Machine-to-machine (M2M) IoT connectivity protocol
• Extremely lightweight publish/subscribe messaging transport
• Open source versions (Mosquitto)
Visualizing Sensor Data in the Database
• No Node-RED options exist (yet) • Our requirements:
– Graphs and/or charts
– Simple (follow the 80/20 rule, just a few knobs) – Flexible
– No Informix timeseries complexity
– Can run on the gateway as well as in the cloud – Open source with “good” license
jQuery
• Cross-platform JavaScript library designed to simplify the client-side scripting of HTML
• Most popular JavaScript library in use today • Free, open-source software licensed under
the MIT License
• Syntax designed to make it easier to navigate a document, select DOM elements,
create animations, handle events, and develop Ajax applications
Ajax
• Initially AJAX (asynchronous JavaScript and XML) • Used on the Client side
• ‘A’ is for asynchronous – data can be sent to the server in the background without interfering with the loading and display of the web page
• Simple data access mechanism uses XMLHttpRequest object
– Despite the name, XML is not required – JSON is often used instead
– PHP data access and manipulation functions also commonly used
Data Access Options
Mongo Client REST Client SQLI Client DRDA Client Informix Spatial Time Series JSON Collection Relational TableData Access Options
MongoDB Client REST Client SQLI Client DRDA Client Informix DBMS Informix NoSQL Listener Informix • NoSQL ↔ SQL Translation• REST, MongoDB Protocol Support
• SQLI, DRDA Protocol Support • Relational, Collection, Time
Series, and Spatial Data Support
Spatial Time Series JSON Collection Relational Table
Data Access Options
MongoDB Client REST Client SQLI Client DRDA Client Informix DBMS Informix NoSQL Listener Informix • NoSQL ↔ SQL Translation• REST, MongoDB Protocol Support
• SQLI, DRDA Protocol Support • Relational, Collection, Time
Series, and Spatial Data Support
Spatial Time Series JSON Collection Relational Table
A REST client is any program capable of
making a HTTP request
Data Access Options
MongoDB Client REST Client SQLI Client DRDA Client Informix DBMS Informix NoSQL Listener Informix • NoSQL ↔ SQL Translation• REST, MongoDB Protocol Support
• SQLI, DRDA Protocol Support • Relational, Collection, Time
Series, and Spatial Data Support
Spatial Time Series JSON Collection Relational Table
You can use the MongoDB PHP
Data Access Options
MongoDB Client REST Client SQLI Client DRDA Client Informix DBMS Informix NoSQL Listener Informix • NoSQL ↔ SQL Translation• REST, MongoDB Protocol Support
• SQLI, DRDA Protocol Support • Relational, Collection, Time
Series, and Spatial Data Support
Spatial Time Series JSON Collection Relational Table
You can also use the Informix SQL or DB2-Informix DRDA
Informix REST - HTTP Method Overview
Method Path Description
POST / Create a new database
POST /db Create a new collection
POST /db/collection Creates a new document
GET / Database listing
GET /db Collection listing
GET /db/collection Query the collection
GET /db/collection?query={…} Query documents that satisfy
query document
DELETE / Drop all databases
DELETE /db Drop a database
DELETE /db/collection Drop a collection
DELETE /db/collection?query={...} Delete documents that satisfy
Informix REST examples
• Query for the databases on localhost:
http://localhost:27018/ Returns…
["city_info","energy_demo","stores_demo","system","test" ,"zipcodes"]
• Query for the people with first name of “Han”:
http://localhost:27018/test_db/people?query={firstName:” Han”}
Returns…
[{"_id":{"$oid":"553017a7e4b0fda5475a51a0"},"firstName": "Han","lastName":"Solo","age":36}]
Informix REST examples
TimeSeries query using Virtual Table:
http://localhost:27018/stores_demo/ts_data_v?query={loc_esi _id:"4727354321000111"}&fields={"loc_esi_id":0,"measure_uni t":0,"direction":0}
Specific loc_esi_id = "4727354321000111" requested. 3 fields (loc_esi_id, measure_unit, direction) to be excluded from the projection list Returns… [{"tstamp":{"$date":1289368800000},"value":0.092},{"tstamp" :{"$date":1289369700000},"value":0.084},{"tstamp":{"$date": 1289370600000},"value":0.09},{"tstamp":{"$date":12893715000 00},"value":0.085},{"tstamp":{"$date":1289372400000},"value ":0.088},{"tstamp":{"$date":1289373300000},"value":0.088},{ "tstamp":{"$date":1289374200000},"value":0.085},{"tstamp":{ "$date":1289375100000},"value":0.091},{"tstamp":{"$date":12 89376000000},"value":0.083},{"tstamp . . .
Informix REST with SQL pass through
http://localhost:27018/stores_demo/system.sql?query= {"$sql":"select tstamp, value from ts_data_v where loc_esi_id=4727354321000111"}
http://localhost:27018/stores_demo/ts_data_v?query={loc_es i_id:"4727354321000111"}&fields={"loc_esi_id":0,"measure_u nit":0,"direction":0}
Informix REST tips
• Data is returned as JSON
• Data is formatted as per MongoDB specs…
– Why? Our REST interface is built on the NoSQL listener and is intended for compatibility with MongoDB.
– This can be a good thing, as JavaScript graphing packages (naturally) rely on JavaScript data types.
Informix REST tips
• Informix TimeSeries uses non standard DATETIME data type
• REST returns DATETIME YEAR TO SECOND (5)
values as MongoDB Timestamps, which are 32-bit unsigned integers with seconds since the epoch (Jan 1, 1970 00:00)
• JavaScript timestamps are represented as
milliseconds since the epoch
• Conversion between MongoDB timestamp and JavaScript timestamps is simple – multiply by
jQuery & Ajax Snippet
$(function (){ $.ajax({ type: 'GET' url: '/api/orders', success: function(data) { console.log('success', data); } }); }); • HTTP GET requestAjax Data Access
• Ajax URL can perform the Informix REST
request using the MongoDB style query or the SQL pass through format
• TimeSeries SQL query can be run against the timeseries virtual table (VTI) or directly against the timeseries
Ajax URL can make Informix REST request
function DoREST( collection , type , params , data , callback ) {
type = type || "GET";
if( params === undefined || params === null ) params = ""; else params = "?" + params;
$.ajax( {
url : "http://" + dbInfo.host + ":" + dbInfo.port + "/" + dbInfo.db + "/" + collection + "/" + params,
type : type, data : data,
contentType : "text/plain; charset=UTF-8", dataType : "json", success : callback, error : AJAXError, xhrFields: { withCredentials: true } } ); }
Access your data with PHP
d3.text("mapread.php", function(txt) { d3.selectAll("#loading").remove(); txt.split("\n").forEach(function(line,i) { line.split(",").forEach(function(d,j) { data[i][j]=parseFloat(d); d3.selectAll(".r"+i+".c"+j).style("fill",function() {return cScale(data[i][j]);}); }) }); })• Use MongoDB, Informix SQLI or DRDA PHP driver • This example shows the JavaScript code calling the
mapread.php function to perform some database
activity and then the inline function operates on the data.
Some Open Source JavaScript Charting packages
Package Link License
dc.js - Dimensional Charting
JavaScript Library http://dc-js.github.io/dc.js/ Apache 2.0 NVD3 Re-usable charts for d3.js http://nvd3.org/ Apache 2.0 Dojo GFX
http://dojotoolkit.org/features/graphics-and-charting BSD
D3.js http://d3js.org/ BSD
YUI Charts http://yuilibrary.com/yui/docs/charts/ BSD jqPlot plugin for the jQuery
framework http://www.jqplot.com/ MIT Chart.js http://www.chartjs.org/ MIT dygraphs http://dygraphs.com/ MIT Flot for jQuery http://www.flotcharts.org/ MIT Flotr2 for HTML5 charts and graphs http://humblesoftware.com/flotr2/documentation MIT gRaphaël http://g.raphaeljs.com/ MIT
D3.js
• The most popular JavaScript library for producing dynamic, interactive data visualizations in web browsers (D3 = Data Driven Documents)
• Does not work real well with jQuery right now, however.
• Most data access is via PHP program
d3.text("mapread.php", function(txt) { d3.selectAll("#loading").remove(); txt.split("\n").forEach(function(line,i) { line.split(",").forEach(function(d,j) { data[i][j]=parseFloat(d); d3.selectAll(".r"+i+".c"+j).style("fill",function() {return cScale(data[i][j]);}); })
Flot
• Flot is a pure JavaScript plotting library for jQuery, with a focus on simple usage, attractive looks and interactive features
• Flot time series data is based on JavaScript
timestamps, that is milliseconds, since January 1, 1970 00:00:00 UTC.
• Timestamps are interpreted according to UTC and, by default, displayed as such. You can set the axis "timezone" option to "browser" to display the
timestamps in the user's timezone, or, if you use timezoneJS, you can specify a time zone
jqPlot
• jqPlot is a plotting and charting plugin for the jQuery
• jqPlot produces line, bar and pie charts
• Computation and drawing of lines, axes, shadows even the grid itself is handled by pluggable
"renderers“
• Hooks into the core jqPlot code allowing for custom event handlers, creation of new plot types, adding canvases to the plot, etc.
Dygraphs
• Basic graph package intended for displaying time series data
• Data must be CSV format, even if returned from XMLHttpRequest function
– Your function would have to reformat the JSON returned from REST request OR
– PHP program to query Informix and format as CSV data
• Built in functionality to display plotted values when hovered over them in the browser
• Nice, built in, functionality to zoom into a range of selected data
Home Grown Visualization
• Can use JavaScript, jQuery and Ajax
• Design your own visualization components based on your requirements
• Determine how to correlate timeseries values with various widget states
• Visualization on Informix and Intel demos take this approach
Home Grown Visualization Example
• “Gauge” shows the number of people on the floor
• Grid show the current location of each person • Display refreshes each second as the count
Summary
• JavaScript & jQuery/Ajax, along with Informix REST interface can make visualization of your timeseries data relatively simple.
• We are working on adding this functionality to our in-progress Gateway Kit
• Look for updates on our GitHub site: https://github.com/IBM-IoT/
