Centralized logging system based on WebSockets protocol

(1)

Centralized logging system based on WebSockets protocol

Radom´ır Sohlich Tomas Bata University Faculty of Applied Informatics Nad Str´anˇemi 4511, 760 05 Zl´ın

Jakub Janoˇst´ık Tomas Bata University Faculty of Applied Informatics Nad Str´anˇemi 4511, 760 05 Zl´ın

Frantiˇsek ˇSpaˇcek Tomas Bata University Faculty of Applied Informatics Nad Str´anˇemi 4511, 760 05 Zl´ın

Abstract:The era of distributed systems and mobile devices brings new challenges in monitoring and controlling the remote components. Watching of components is usually realized through log records. To obtain comprehensive view of distributed system the centralization of logged information is usually required . There are many central-ized log solutions such as Syslog,Greylog2,Logstash or cloud service Loggly that implement the functionality of gathering log messages and data from remote components and devices. These solutions are generally based on one way data transfer, that directs from client to server. The simplest solutions use basically log file synchronization to obtain data from remote components. More sophisticated solutions use periodic reading of remote system web service or expose other protocol endpoints like syslog protocol. This research paper proposes centralized logging solution based on Websocket technology. In section 4 article describes features, architecture and communication scheme. Section 5 compares the proposed solution with existing applications. The last section 6 discusses the future work and enhancements of the proposed system.

Key–Words:Centralized logging, Log4j, WebSockets, Syslog, Greylog2

1 Introduction

The era of emerging software with distributed archi-tecture emphasizes difficulties with monitoring and analyzing functionality of remote components. The simplest way how to track behavior of system ele-ments is logging its operations during the runtime. These data then bring the record of program flow and also the information describing system failure or mal-function.

The trivial logging solution is that the data is writ-ten to local storage. This way is sufficient if the whole system is located on the same machine. The problem occurs if the system is located across multiple devices and the components write logs to local files. In this case the information of whole system behavior is lo-cated in separate files and these must be merged and analyzed.

There are two general approaches of solving the problem of distributed system logging. Both are based on centralizing information on single machine. These approaches differ in the way how the data are col-lected.

The first technique is that the components of sys-tem write log record to their local storage and in the system there is a subsystem which periodically syn-chronizing its log storage with the remote component. Alternatively the log server sends the request to

spec-ified source and receives the information from that source.

The shortcoming of the solution is that the en-tire log file needs to be synchronized and the log records are not present in centralized component in real-time. The second solution stands on exposing re-ceiver for communication with specific protocol. The remote component then sends the log messages di-rectly to log server. Alternatively the remote compo-nent could contain thick logging client, which con-nects directly to remote storage(e.g. database). In both approaches the log server is just passive receiver for log data. This paper proposes experimental imple-mentation based on the second approach with some enhancements in server functionality. The solution is build on lightweight WebSocket technology, NoSQL storage and Java application server. The main im-provement is in the usage of WebSocket communica-tion, not only to send log data, but even to control the client settings and functionality.

Organization of paperSection 3 describes the re-quirements and general description of solution. Fur-ther in section 4 the architecture and technologies are described. The section 5 contains comparison with another centralized logging solution. Last section 6 summarizes the results of testing and discusses future work.

(2)

2 Related work

The area of this problem is fairly covered so the study of existing solutions were done (generally Java plat-form implementations). There are some widely used systems and libraries using one of the mentioned ap-proaches. Greylog2 [11] is log capture and analyzing tool. It has a flexible input types, including syslog, plaintext, and GELF. Additionally it is able to read from HTTP API. Greylog2 using MongoDB[12] as a storage and Elasticsearch[13] to analyze and search through the log records.

Another related solution is the Syslog-ng[14]. It supports client-server mode, which is based on con-figuring one instance of Syslog-ng on client machine to transfer log messages to server machine through specified channel (e.g. udp,tcp connection or syslog protocol). Also syslog-ng driver can be used to write messages directly to remote storage (e.g. SQL stor-age,NoSQL storage). Syslog-ng doesn’t provide log analysis tool, this feature must be realized through third party tool.

Logback brings very similar concept to Graylog2, but does not provide complete functionality for log analysis. To store and analyze logs, Elasticsearch must be integrated.

Loggly is a commercial cloud service, commonly called ”logging as service”. The service is capable of gathering logs from every popular programing lan-guage or platform and the data could be sent using almost every protocol (Syslog TCP, Syslog UDP, Sys-log TCP w/ TLS, or HTTP/S). The disadvantage of this solution is that the system must be connected di-rectly or indidi-rectly (over proxy) to the Internet.

3 Requirements

The analysis of the related projects reveals the main requirements for proposed implementation.

• multi-platform (Linux,Windows) server solution

• flexible NoSQL storage for log records

• user friendly web interface

• access through REST API

• client transfer protocol widely supported across commonly used programming languages

• lightweight client implementation

• easily implementable message format

• simple configuration from server side

• open-source

4 Architecture

The high level architecture is very simple and it is based on client-server model.[2] The server side con-sist of application that receives and processes logs, ap-plication server and persistent layer.

Fig. 1: Architecture design

4.1 Communication

As the solution required communication in both direc-tions(client to server, server to client), suitable tech-nology had to be selected. To ensure simplicity and versatility, a web based protocol is preferred. There were designed many two way communication proto-cols that use HTTP transport layer to benefit from existing infrastructure (authentication, secure trans-port, proxies). However these protocols are trade-offs between efficiency and reliability as the HTTP protocol is not initially designed for bidirectional communication[6]. As the substitution for these trade-offs, WebSocket protocol was designed. The protocol uses the HTTP transport layer as is and it is designed to work on standard port 80 or 443 for secure trans-port. After a micro-benchmarks between HTTP alter-natives and WebSocket protocol, the WebSocket tech-nology was selected. One of the advantages of Web-Socket protocol is that it uses one TCP connection for the communication and avoids the repetitive open-ing of connection, which reduces the performance. Same as basic HTTP protocol the WebSocket proto-col has wide support across programming platforms, so the implementation of clients for various platforms is possible. The log messages are JSON formated and sent by WebSocket text frame to/from client. The JSON format was chosen for its flexibility and sup-port in many programming languages. The JSON log

(3)

message contains all standard fields common for log-ging. There is also field for arbitrary object to be logged. This feature simplifies the data-mining opera-tions from log records. The remote reconfiguration of logging client is implemented via text frames in spe-cial format different from standard log message. Also the direction is from server to client. The idea be-hind this feature is that the server could remotely con-trol settings of each client log level or identification of component.

The communication scheme on fig.2 shows the entire process of establishing connection and message exchange. After the WebSocket handshake, server sends initial configuration message to client , which contains the information about log level (in this case FINE) and identification of component, if it is precon-figured by log server admin. After this information exchange, the client sends the log messages with ap-propriate level. The reconfiguration message shows how the remote setting of log level is done (in this case INFO level).

Fig. 2: Communication scheme

4.2 Server

The server part is Java Enterprise application, which is running on Wildfly[3] application server. The ap-plication implements WebSocket endpoint for logging clients. Server contains remote control logic, user in-terface and additional REST API to access the func-tionality designed for log analysis and client remote control. User interface consists of configuration of clients, log analysis and search engine. The persis-tent layer is based on MongoDB NoSQL database. It was chosen for its flexibility and also it could be eas-ily integrated with advanced indexing, searching and analyzing tools (e.g Elasticsearch, Kibana, Hadoop). The solution transfered the logic of log message writ-ing and processwrit-ing to server side. Server

implemen-tation uses MongoDB Java driver to write logs and to process the log messages asynchronously. The asyn-chronous writing brings the increase in throughput.

4.3 Client

Thanks to WebSocket technology, the imple-mentation of client is possible in various lan-guages(C++,.NET,Java,JavaScript,Python and others). The experimental client is implemented in Java programming language using the Jetty Web-Socket Client API implementation. Serialization of LogMessage is implemented by Jackson library. If the connection to log server is not present, the client caches records and after the connection is established again, it sends all cached logs to server.

5 Comparison

To test proposed implementation against an existing solution, the log4j2 NoSQL appender was chosen as the nearest matching solution. This comparison mea-sures the performance of logging clients, where log4j2 NoSQL appender uses MongoDB Wire Protocol to transfer serialized messages. The custom client uses WebSocket protocol as described above. The method-ology of comparison is as follows:

• create a logger object

• insert k log records (text logs, logs with excep-tion)

• measure duration of insert operation

The benchmark is implemented also as Java applica-tion, as the Log4j2 is Java library. The measurements were realized on clear database collection and in sepa-rate runs. Every measurement was repeated 40 times. The insertion of 1000 log records was chosen as most representative sample size if we consider, that com-mon application does not insert more than hundreds of log by one Logger instance.

In case if there is no additional object(exception) to serialize the proposed solution shows higher aver-age time to insert 1000 logs. Figure 3 shows the com-parison of average duration of 1000 info log messages insertion. The measured value of experimental imple-mentation is almost similar to log4j appender. Figure 4 displays the average duration of inserting 1000 log record containing exception object. In this case the ex-perimental implementation achieved lower time value. This result is caused by more simple implementation of exception serialization and also by transferring of persistence operations to server. Also the average du-ration is nearly constant.

(4)

Fig. 3: Average duration of inserting 1000 logs with-out exception

Fig. 4: Average duration of inserting 1000 logs with exception

6 Conslusion and future work

As described in paper, there are wide array of central-ized logging solutions. From simplest solution of file replication to sophisticated cloud services like Loggly. We proposed a centralized logging that benefits from WebSocket protocol as widely supported so-lution of bidirectional communication. The proto-col also uses existing infrastructure. The experi-mental solution is based on Java platform, but the clients could be implemented in other programming languages. The solution was compared with exist-ing implementation of Log4j NoSql appender. The benchmark of proposed solution proofs, that even not optimized version of that implementation is compa-rable to existing widely used Log4j2 appender. The tests also display, that the time to send a log record remains stable if the log record contain an object of exception. On the other hand the comparison also re-veals that there is space for optimization. The serial-ization process of log message could be improved as it creates a performance leak of whole system. For

Java, Python and C++ Protocol buffers[15] could be solution, but the usage of this technology eliminates the versatility of message format. There are also new opportunities to explore in way of remote configura-tion and client funcconfigura-tionality control. In proposed sys-tem the reconfiguration of log level and component name are implemented, but further attributes and even remote functions could be added e.g. gathering infor-mation about remote system (utilization,source usage) dependent on client platform.

References:

[1] RFC6455. The WebSocket

Proto-col. 2011.: Internet Engineering Task

Force (IETF), 2011. Available from:

https://tools.ietf.org/html/rfc6455

[2] BERSON, Alex. Client-server architecture. McGraw-Hill, 1992.

[3] Wildfly [online]. 2013 [cit. 2014-10-29]. Avail-able from: http://wildfly.org/

[4] Mozilla Developer Network: WebSockets [online]. 2014 [cit. 2014-10-29]. Avail-able from: https://developer.mozilla.org/en-US/docs/WebSockets

[5] Qt Project: Qt WebSockets C++ Classes [on-line]. 2014 [cit. 2014-10-29]. Available from: http://qt-project.org/doc/qt-5/qtwebsockets-module.html

[6] RFC6202. Known Issues and Best Practices for the Use of Long Polling and Stream-ing in Bidirectional HTTP. University of Rome ”Tor Vergata”: Internet Engineering Task Force (IETF), 2011. Available from: https://tools.ietf.org/html/rfc6202

[7] CROCKFORD, Douglas. The application/json media type for javascript object notation (json). 2006.

[8] ABUBAKAR, Yusuf; ADEYI, ThankGod S.; AUTA, Ibrahim Gambo. Performance Evalua-tion of NoSQL Systems using YCSB in a re-source Austere Environment. Performance Eval-uation, 2014, 7.8.

[9] The State of Logging in Java 2013. In: VAN CAMP, Balder. Zeroturnaround [on-line]. 2013 [cit. 2014-10-29]. Available from: http://zeroturnaround.com/rebellabs/the-state-of-logging-in-java-2013/

[10] APACHE SOFTWARE

FOUNDA-TION. Apache Log4j 2 [online].

2014 [cit. 2014-10-29].Available from: http://logging.apache.org/log4j/2.x

(5)

[11] TORCH GMBH - THE GRAYLOG2

COMPANY. GRAYLOG2 [online].

2014 [cit. 2014-10-29]. Available from: http://www.graylog2.org/

[12] MONGODB, Inc. MongoDB [online].

2014 [cit. 2014-10-29]. Available from: http://www.mongodb.com/

[13] ELASTICSEARCH BV. Elasticsearch [on-line]. 2014 [cit. 2014-10-29]. Available from: http://www.elasticsearch.org

[14] BALABIT IT SECURITY. Syslog-ng:

The Foundation of Log Management [on-line]. 2014 [cit. 2014-10-29]. Available from: http://www.balabit.com/network-security/syslog-ng

[15] GOOGLE, Inc. Google

Develop-ers: Protocol Buffers [online]. 2014

[cit. 2014-10-29]. Available from: