4. Implementation
4.4. Application development
4.4.1. Implementation of data endpoint
In order to receive a data from the manufacturing facility, a starting point inside of application server is created, which acts as a gateway for stream of incoming data.
Every message from Event Hub is sent over HTTP as POST message and it is consumed by endpoint, where input message is transformed to XML Node format and then redirected to CEP engine. In case of success, server responds with HTTP 200 – a standard response for successful HTTP request. If error occurs when sending event to Esper engine, server will respond with HTTP 500 Internal Server Error. The case of successful message delivery is illustrated on Figure 20.
Jetty Endpoint CEP Engine
Event Hub
HTTP POST XML message
HTTP STATUS 200
Convert to XML node
Send XML node event
Figure 20: Message delivery to application 4.4.2. Implementation of data processing
From the endpoint data is delivered to Esper CEP engine, where it is processed and prepared for storing.
In order to make CEP engine aware of incoming messages, a configuration file esper.cfg.xml is used. This file configures Esper by defining types of existing messages.
If incoming XML message is not predefined, an exception will be thrown by Esper engine, stating that the message is not configured.
Figure 21 illustrates the workflow of CEP engine. When XML event enters the engine, it matches the message to relevant rules. If the rule is satisfied, subscriber is triggered, which is responsible for processing of the data by getting relevant information from the message and passing it to data access layer. If XML event doesn’t satisfy the rule, it is sent to memory of engine, where all messages for specific rule are aggregated until the rule is satisfied.
CEP Engine
XML event Enters
EPL rules Matches
if Rule satisfied?
Subscriber Yes
Activates Data access layer Passes data
Database Persists
Memory
No Aggregates event
Figure 21: CEP engine workflow CEP engine has two main functionalities:
Parsing of the incoming messages
It is achieved by having an EPL rule, which selects all attributes of message every time new message arrives, with subscriber receiving Java objects containing values of attributes
Calculation of KPIs
KPIs are calculated in a real-time according to existing EPL rules
4.4.3. Development of Data Access Layer
Data access layer is an intermediate layer between application logics and database.
It provides a loose coupling for fetching data from database, by offering an interface with methods that are responsible for database queries, and separating it from application logics.
An XML configuration file persistence.xml is used for specifying database connection details, which is placed under /META-INF folder. The file is demonstrated on a Listing 3. It contains desired name of persistence unit (database cluster), list of mapped to database tables Java classes and connection-specific parameters. Any amount of persistence units can be added to the file. Java classes (called Entities), intended for
mapping to database tables, have @Entity annotation, which has a property referencing to relevant persistence unit.
<persistence-unit name="persistenceUnit">
<class>com.model.MessageModel</class>
<class>com.model.MessageModelKey</class>
<properties>
<property name="kundera.nodes" value="localhost"/>
<property name="kundera.port" value="9160" />
<property name="kundera.keyspace" value="name" />
<property name="kundera.dialect" value="cassandra" />
<property name="kundera.client.lookup.class"
value="com.impetus.client.cassandra.thrift.ThriftClientFact ory" />
<property name="kundera.cache.provider.class"
value="com.impetus.kundera.cache.ehcache.EhCacheProvider"/>
</properties>
</persistence-unit>
Listing 3: Database connection configuration file
Physical connection to database is made with an instance of EntityManager, which provides functionality to perform CRUD (Create, Update and Delete) operations on a database. An independent EntityManager is obtained for every single request from an EntityManagerFactory. EntityManagerFactory and EntityManager interfaces are available as a part of JPA 2.0. EntityManagerFactory is a heavyweight object, therefore it is instantiated only once during application start and is stored inside of Spring Container, so that EntityManager can be easily created at any time. EntityManager is injected into objects with @PersistenceContext annotation.
Queries for data access are performed with JPQL, which is wrapped by Kundera.
The example of query formulation is illustrated on a Listing 4. This query selects some specific data, which is mapped to database with EnergyData entity, and limits search to particular cell and time range.
String query = “SELECT e.data from EnergyData e where e.key.cellId = :cellId
and e.key.timestamp > :timeStart and e.key.timestamp < :timeEnd”
Listing 4: JPQL query example
Query is using named parameters (“:cellId”,”:timeStart”,”:timeEnd”) in order to provide a flexibility in accessing the data. These parameters are further evaluated according to specific user request. After all, Query object is created, to which textual representation of query and named parameters are passed. Data can be fetched from database by calling a getResultList method of Query object.
4.4.4. Development of API SOAP Web Services
API based on SOAP is implemented with Spring Web Services, provides various methods for accessing data in the database. Spring Web Service is a contract-first service, which means that XML schema needs to be created before development of service logics in Java. The schema contains description of methods used to access data, consisting of request and response messages, as it is illustrated in example on Listing 5.
<xs:element name=”DataRequest”
<xs:complexType>
<xs:sequence>
<xs:element name=”cellId” type=”xs:int” />
<xs:element name=”timeStart” type=”xs:long” />
<xs:element name=”timeEnd” type=”xs:long” />
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name=”DataResponse”
<xs:complexType>
<xs:sequence>
<xs:element name=”cellId” type=”xs:int” />
<xs:element name=”value” type=”xs:float” />
<xs:element name=”timestamp” type=”xs:long” />
</xs:sequence>
</xs:complexType>
</xs:element>
Listing 5: Definition of data access request and response
Based on this schema and configuration file, which defines host of Web Service, JAXB mapping classes for serialization and deserialization of incoming XML requests, Spring automatically generates WSDL file, which becomes available for clients. The next step after contract definition is development of Java Web Service endpoints, which are mapped to relevant requests in WSDL with help of annotations, where request processing is performed. Further, endpoint makes a query to database through data access layer, prepares response message and sends data back to client. The whole cycle of operations and interaction between layers of application from SOAP request to SOAP response is depicted on a Figure 22.
Third Party Application JAXB marshaller DAO Cassandra SOAP request
Binding to JAXB class WS endpoint
JAXB class
Request object
Data request
Query
Query result
Map to POJO
List of POJOs Data response
Process response
Final data JAXB response object
Binding to SOAP
SOAP message SOAP response
Figure 22: Web Service workflow
REST Web Services
RESTful API is developed with web application framework Pedestal for Clojure.
Service is deployed with the help of build automation tool Leiningen. API provides flexible methods for reading historical data from the database. These services are intended for web applications, which need to present historical data by creating graphs and charts or analyzing the data.
Clojure application is running as a separate server application, receiving HTTP requests. For every request a mapping logic is defined, stating which function needs to be executed when request arrives. The function is responsible for handling of request by creation of relevant query to the database, processing the response from database to correspond to required response format, and returns JSON object back to the client. The whole workflow sequence is presented on a Figure 23.
Third Party Application Routing Handler Mapped Function Data Access Function Database
JSON request
Request parameters
Request parameters
Query
List of data List of data
Manipulations on data
Response List
JSON response
Figure 23: REST Web Service workflow
The processing stage can be performed very fast even on the very large data set, because the data is presented in form of a collection, which can be easily manipulated with Clojure.
4.4.5. Development of KPI definition interface
Due to the fact that CEP is one of the core functionalities of the information warehouse system, access to event processing rules for responsible personal needs to be transparent. KPI definition interface is created to satisfy this requirement. The interface is developed with JavaScript and HTML5.
The interface, presented on Figure 24, consists of 3 areas:
1. Rule creation
User can add new rule and give it a name for identification purposes 2. Rule selection, display and editing
User can navigate through already existing rules and select them for editing or deleting
3. Message references
This area provides a reference to existing messages in the facility
Figure 24: KPI definition interface
The interface is intended for a personal, which is familiar with EPL rules syntax.
4.4.6. Web application configuration
All modules described previously are initialized with spring configuration file web.xml, which is responsible for initialization of context configuration files and Web Service mappings. There are two context configuration files:
Application context
responsible for initialization of CEP engine, HTTP endpoint and Entity Manager Factory
Spring-ws-servlet
responsible for initialization of Spring Web Service
Figure 25 demonstrates the configuration and initialization process of the application.
Web application config
Figure 25: Application configuration with Spring
After configuration is completed, the application is ready to receive and process incoming data and Web Service requests.
4.5. Development of OSGi module
Application also needs to be made accessible inside of OSGi infrastructure. In order to complete the integration, a module responsible for access to Cassandra database needs to be created. This module can be used for generation of services, which can be used by other OSGi modules to access data from the database. However, the major challenge faced during this step was inability of Kundera library to run in OSGi environment. This is explained by the fact that Kundera is based on Thread context class loading, meaning that its context can’t be loaded to OSGi thread, from where it could be accessible by other modules.
This problem was solved by creation of a new module, called kunderaLibrary, which exports all dependencies required by Kundera, and creates new methods to access the interface of Kundera library. The module contains all Java Archive (JAR) files, which are referenced by Kundera, resulting in a very heavy weight of the final module (34MB). In a bigger application heavy-weight module can reduce performance of the system, but this was the only possible solution to integrate Kundera library to OSGi environment.
5. RESULTS
5.1. Implementation test-bed
Designed approach system was implemented for a manufacturing line located in premises of FAST Laboratory in Tampere University of Technology. The line is composed of 12 cells, equipped with robots and conveyors (Figure 26), which is capable of performing simulation of mobile phone manufacturing by drawing parts of phone on a paper.
Figure 26: FAST manufacturing line
The line is equipped with S1000 controllers and E10 energy analyzers. S1000 controller is a programmable Smart Remote Terminal Unit device, designed to operate in industrial environment and compliant with all industrial signal types and levels, which provides web-based Human-Machine Interface (HMI) and Web Services integration. E10 is an additional module for S1000, which analyzes 3-phase electrical power consumption [80]. The following parameters are analyzed in E10:
Root mean square (RMS) voltage
RMS current
Active, reactive and apparent power
Active, reactive and apparent energy
5.1.1. Manufacturing line events
There are 3 types of events generated in a factory floor: energy message, equipment change state message and notification message.
Energy Meter message contains energy consumption related information of each cell, divided by 3 energy phases – A (robot), B (controller) and C (conveyor), as it is presented in example of message on Listing 6.
<EnergyMeter xmlns="http://www.tut.fi/fast/energymeter"
xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing"
AIRMS="0.26" AVA="65.81" AVAHR="1255" AVAR="-62.84" AVARHR="-1217"
AVRMS="242.33" AWATT="-6.90" AWATTHR="-128" BIRMS="0.31"
BVA="77.71" BVAHR="1486" BVAR="-66.81" BVARHR="-1298"
BVRMS="239.99"
BWATT="5.48" BWATTHR="111" CIRMS="0.26" CVA="63.80" CVAHR="1222"
CVAR="-61.06" CVARHR="-1184" CVRMS="240.92" CWATT="-9.90"
CWATTHR="-190" LINEFREQ="50.06" cellID="12"
dateTime="2000-01-02T21:24:16.490"
eventId="0" eventName="scan_energy_measure" src="energyMeter"/>
Listing 6: Energy Meter XML message example
Equipment Change State message carries data about status of robot inside of each cell, specifying current and previous CAMX (Computer Aided Manufacturing using XML) states, as well as all relevant information, which is presented on a Listing 7.
<EquipmentChangeState xmlns="http://www.tut.fi/fast/robot"
xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing"
cellID="6" condition="NORMAL" currentState="READY-IDLE-BLOCKED"
dateTime="2000-01-04T00:18:41.970" eventID="xx"
eventName="ItemWorkComplete"
palletID="13" previousState="READY-PROCESSING-EXECUTING"
recipeNum="7"
src="robot" toolID="1" transID="2"/>
Listing 7: Equipment Change State XML message
Notification message contains status of conveyor and relevant to it information for each of the cells (Listing 8).
<NotificationMessage
xmlns="http://www.pe.tut.fi/fast/wsdl/ConveyorService"
xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing"
cellID="5" dateTime="2000-01-03T23:49:41.STOPPED" eventID="75"
eventName="palDeparture" fromZoneID="4" opResult="NA"
palletID="13" src="conveyor" toZoneID="5" transID="NA"/>
Listing 8: Notification Message XML message
5.1.2. Energy Meter data model
According to message structure, described in previous section, contents of Energy Meter message can be divided into phase-specific energy consumption data and common cell data. The potential query use cases are presented below:
Get active energy power data for robot in cell number 1 during previous day
Get energy consumption for conveyors in each cell during last week
Get specific energy parameters of all phases in several cells during specific time range
From these use cases a conclusion can be made that data can be identified distinctly by cell number, energy phase and time. These three parameters define a compound key, where cell id is a partition key, energy phase and timestamp are clustering keys.
According to this data model, energy data will be stored on different nodes of cluster based on a hash value of cell id, and will be clustered in accordance to energy phase and will be sorted by timestamp. Figure 27 illustrates the data model in an understandable way and shows the way data is clustered.
CELL ID
Figure 27: Data model of Energy Meter table
5.1.3. Equipment Change State and Conveyor Notification data models As it was described earlier, equipment change state messages contain relevant information about status of robot in each cell. This information is tightly connected with energy efficiency, thus needs also to be stored and available for requests.
Potential query use cases look like as following:
Get amount of occurrences of “READY-PROCESSING-EXECUTING” state in cell number 1 during last hour
Get timestamps of events when robot was blocked in cell number 3
According to presented use cases and structure of XML message, it can be determined that equipment change state data can be uniquely identified by combination of cell id, event name and timestamp.
Conveyor Notification message also carries information that might be relevant for analysis of energy efficiency of manufacturing line. The query and data models are similar to Equipment Change State message. Illustrations of Equipment Change State and Conveyor Notification tables are available in Appendix 2.
5.1.4. KPI data model
KPI data is also needs to be stored in a database with predefined data model.
However, in contrast to described earlier tables for data coming from devices, a single data model needs to be defined for any type of KPI. This approach will result in a possibility to focus in the future on development of KPIs without requirements for creating new data structure for each newly introduced KPI.
All information in a context of KPI can be divided into 3 types with following contents:
KPI definition o KPI name
o Cell ID (or consumer/scope) o Timestamp
Actual value of KPI o Numerical value
Metadata
o Pallet ID o CAMX State o Product
o Any other relevant information
According to these types of data, query use cases can be defined:
Get values of energy consumption KPI for cell number one for last month
Get values of energy consumption KPI for whole manufacturing line for specified time period
Get value of energy consumption KPI and its metadata of specific cell for specified time period
The data model of KPI table thus should have a compound key, consisting of KPI name as partition key, cell ID and timestamp as clustering keys. Moreover, KPI metadata can include any relevant for particular KPI information, thus data model needs to support a flexible addition of KPI metadata parameters. This can be achieved with help of collections support realized in Cassandra, so that metadata column can be
defined as map containing key-value pairs of text variables. Key-value pairs can be added when needed for specific KPI and can always be fetched, when requesting the data. The data model illustration is available in Appendix 2.
5.2. Application integration
In order to complete the thesis work, the application has been integrated to the FASTory production line. Cassandra database has been deployed to Linux server, which has accessible public IP. Cassandra has also been configured to be bounded to the public IP of the server.
The application has been running on a Windows machine connected to Local Area Network (LAN) of the production line, so that messages from Event Hub could be redirected to the application.
Application has been configured to access the deployed database, and all used entities for mapping of data has been added, as it is illustrated on a Listing 9.
<persistence-unit name="fastoryDatastore_pu">
Listing 9: Database configuration file persistence.xml
Http endpoint inside of the application has also been configured to listen to incoming HTTP POST requests on the IP address of local machine. This address was also added to configuration of routes inside of the Event Hub.
5.3. Web services
In order to make stored in database data transparent and available for relevant personal, Web services for data access have been developed. All created services are summarized in a Table 16.
Table 16: Developed Web Services
Name Functionality Type
Historical Data Returns requested historical data SOAP
KPI Data Returns requested KPI data SOAP
Full Energy Data Returns historical data of energy meter message REST Full Robot Data Returns historical data of robot equipment change
state message
REST
Full Conveyor Data
Returns historical data of conveyor notification message
REST
Chart data Returns historical energy data, which can be used for plotting charts and graphs
REST
5.3.1. Historical data
This service is used to access any available historical energy data in a various combinations. Inputs of the service include:
List of cells
List of phases
List of energy parameters
List of time ranges
In other words, this service allows user to form as flexible and detailed request for
In other words, this service allows user to form as flexible and detailed request for