Description of the benchmarking programs

The following subsections explain how the performance was calculated for the messaging solutions as well as the details regarding the APF evaluation.

7.1.1. Messaging solutions performance evaluation

After covering the general description of the steps that were taken in order to complete the evaluation, the following subsection will try to elaborate the approach taken for the programs. In the first step, the benchmarking programs follow a pattern of measuring each solution at particular points. The tests were done for both ActiveMQ and RabbitMQ and the messaging solutions were used as is, i.e. with the default configurations and no additional changes to the configurations were made for improving the performance.

There is a single producer and consumer in each of the scenarios sending/receiving messages over a single queue. The messages are auto acknowledged upon receipt by the consumer and the size of the messages is always constant at 13 Bytes. There is awareness that, in actual production environment the message could be variable and larger. As discussed with Aurenav LLC, the message size, when the solution was deployed in the banking and financial sectors, there was a noticable difference, starting from several K to several hundred K messages. What varies is the number of messages that need to be exchanged, the number of computers used and if persistence mode is enabled. In regards to the performance of the messaging solution, three main variables were considered:

 The throughput which the producer/consumer achieved, measured in messages per second.

 The total time it took from sending the first message until the last message was received.

 Last, at the consumer side, the average latency for the delivery of the messages was measured.

The generic formula used for calculating the throughput was dividing the number of messages that were sent by the time it took for their transfer. The throughput was measured at two different points: one point was at each second to follow the overall performance during execution time and one at the end of the performance test for reporting purposes. For the first scenario, whenever a one second trigger was started, it was calculated how many messages have passed within a time interval. Thus there are the following four variables:

 – an increasing counter representing the latest number of messages sent/received at the time of reporting

 – an increasing counter representing the number of messages sent/received at the previous reporting time

 - time marked at the moment of reporting  - time of the last report

80

In addition, the throughput can be calculated as seen on the formula below:

In addition, at the end of the test, the final throughput is calculated in a similar manner, with the notable difference that the total number of messages was used over the total time it took from the start to the end of the test. Last but not least, the latencies were saved for each of the messages so that they would be compared. The latency for each message was calculated by having the difference between the time marked at sending of the message and the time marked at receiving it at the consumer side. This helped in evaluating the average latency which was calculated in the following manner. First of all if there is:

 - Time marked at sending a message  - Time marked at receiving a message

Then the latency can be calculated in the following manner:

It should be noted that all the data processing was performed after the execution time of the tests so it does not influence the performance.

7.1.2. APF performance evaluation

Besides the benchmarking tests of the message solutions, the APF was adjusted to facilitate the evaluation process as well. For this purpose, there were four identified measuring points within the framework for both the requester and the provider. As it was previously mentioned in section 6.3.:

 Requester is the application that generates a request and sends it off to another APF instance, from which later on it receives a reply;

 Provider is the application that waits for request and based on the received request, generates a reply that sends it off to the application, from which it received the request in the first place;

Within the requester, the first point is noted at the beginning, when the request is received in the AIC component. Second point is when the request has reached the final MPAS component before putting the message on the channel. As it was mentioned, the actual delivery and exchange of the messages was not measured at this step. Rather, when a reply message was received the third point marked the time of the message arrival within the MPAS component. Last but not least, the fourth point was when the reply reaches the AIC component, before delivering it to the application.

Similarly, it was done for the provider, by having again four measuring points. The first point is at the receipt of the request by the producer. The second point is at the AIC

81

component level to measure the time taken for the request to be processed and delivered to the application. There is another division here to avoid measuring the processing time necessary for preparing the reply. The third point is from the AIC component when the reply message is being sent through the components. The last point is in the MPAS component, before putting the reply on the channels.

In addition, to the tests described previously, the evaluation process went into more details of testing the actual performance of the individual components that comprise the APF. For this matter, similarly to the previous description, when a message entered a particular component, the time was marked, as well as when the message left the component. The measurement points were set for each of the components, namely AIC, TVE and MPAS. These measurement points would mark the start and completion time for the message to pass through the component giving valuable insight into the performance of the individual components.