Consumer device capabilities and limitations
8.4 The move toward a query language
function webSocket() {
if ("WebSocket" in window) { // open a web socket
var ws = new WebSocket("ws://<url to service>");
ws.onmessage = function (event) {
document.getElementById("event").innerHTML += event.data + <br>";
};
} }
Listings 8.1 through 8.3 should give you a sense for how each of the integrations would work. Both server-sent events and WebSockets work on a message-by-message basis, allowing you to easily and cleanly apply your business logic. Arguably, the Long Polling does as well, but remember, with the Long Polling example it’s the browser that’s initiating the request by setting a timeout to execute the request each time.
Regardless of which technology is used by a streaming API, you should be able to integrate with it using JavaScript and a browser. Chapter 9 walks through this in more detail, and we’ll build out a client application.
8.4 The move toward a query language
Shortly after you have your integration working, whether you’re building a web-based streaming client or a client in your favorite programing language, your users are going to ask for the ability to apply filters and other criteria to the data stream.
They’ll want to be able to query it like they are used to querying traditional data stores. This may not happen at first— they will likely be ecstatic to have a stream of data flowing and to learn things about their business that they never knew. Quickly, though, the novelty will wear off, and the requests will come in. You’ll be in good company because this is where streaming is going, and a lot of work is being done
Listing 8.2 HTTP Long Polling
Listing 8.3 WebSockets
web socket server The function called every time a new message is received
Updates the “event”
element on the web page with the new data received
See chapter 9
figures 9.6 and 9.7
in the industry in this direction. Unfortunately, today you may have to do some of this client-side; even though all the stream-processing engines have or are adding SQL-like capabilities to them, the SQL support in third-party streaming APIs will vary wildly.
The introduction of SQL to a streaming client is interesting. If you’ve used other business systems, you’ve certainly used SQL. It’s one of the most ubiquitous data access models there is. If the streaming API you’re using provides SQL or support for a different query language, then for your client you may be able to get away with pro-viding a thin layer over that. In essence, you offer the same support as the API and pass the query down to the API. But if the streaming API you’re using doesn’t port any query language, and you need to provide SQL or other query language sup-port, there’s only a certain amount you can do in the client. To properly provide query support you’ll want to either involve the team that is building the streaming API you’re using or, if it’s a third party, consider building your own API that is a proxy to that one. Before going into the details as to why, look at the architecture illustrated in figure 8.12.
With this architecture in place we can provide complex query capabilities in our browser-based application. All we need to do is to communicate with the proxy streaming API as if it were the real streaming API. Then in the browser we can expose a way for a user to enter a SQL query, pass it back to the proxy, and have the proxy execute the query as the stream is flowing.
If you find yourself in this situation and you need to implement the streaming API proxy to provide SQL query capabilities to your users, consider using Apache Calcite (https://calcite.apache.org) for this purpose. It’s a library specifically designed for processing SQL with data in a stream or at rest. There may be a Calcite adapter for your data source, but if not it’s designed with extensibility in mind and should be easy to adapt to your needs.
8.5 Summary
In this chapter we looked at concepts to keep in mind when building a streaming cli-ent. We covered quite a bit of material that probably went deeper than you were
Proxy streaming Events API
3rd Party streaming
data API Apply SQL
On receipt of the message from the streaming API we need to apply the SQL query.
Events
Web UI
Figure 8.12 High-level architecture of proxy streaming API
expecting for a chapter on the client side of a stream. Some of the concepts covered here were seen before in chapters leading up to this.
The chapter covered the following:
The core concepts to think about when delivering a stream of data
The quest to query a stream
The key take-away from this chapter is that there is a lot more to think about when delivering streaming to users than a flashy dashboard. At a certain point, users will want to query it, and many other times you’ll want to deliver the stream to a third-party application. At those points, the art of delivering a stream to a client takes on a whole new life of its own.
Chapter 9 puts this all into practice—we’ll be building a complete streaming system.