In order to determine the performance of the CSB–UCC during peak loads (i.e., increasing work demand), the scalability test is conducted. In this work, the workload represents the increasing number of mobile users’ requests in the system. This experiment requires heavy users and deviates from the previous experiments in terms of the required numbers to determine the actual workload capacity. Thus, a load generating tool, which is built in Erlang and identical to the Apache Bench software (http://httpd.apache.org/docs/2.2/programs/ab.html), is used as the client to send concurrent HTTP requests to the CSB–UCC for resources at a controlled rate. The load generating tool is installed on the windows 7 enterprise edition server with the specifications in section 4.1 in the laboratory. The set–up for the load capacity testing of the CSB–UCC is illustrated in Figure 4.16.
Figure 4.16: Set–up for the Scalability Test
The computer on which the load generator is running connects to the controller component of the CSB– UCC through a Gigabit Ethernet connection. The load generator is configured to simulate the activities of concurrent users of the system in multiples of 30000. The total number of concurrent HTTP requests
that the users can send ranges from 10000 to 1000000. The load generator is configured to return the mean throughput (i.e., the rate at which jobs are completed, measured in request per second) every 10 seconds. In the experiment, the throughput is observed for six different set–ups where the system is centralized (i.e., only 1 broker) and distributed where the brokers are increased from 2 brokers to 6 brokers. Each of the broker (which is the same as sub–proxy) is deployed on an Amazon EC2 instance in North Carolina.
The point here is to get an idea about how the CSB–UCC will behave in different workload scenarios whether centralized or distributed. When the system is distributed, the request handling tasks include the delegation of the load to the various sub–proxies (or brokers) by the controller. The load generator is configured to send concurrent HTTP requests to the CSB–UCC to consume the data that reside on the CouchDB database (as described in section 4.2.3). This experiment is conducted with the data payload from the Clandestine Anomaly Game. It is important to state that the size of the data is not under consideration here because the experiment is not retrieving the actual data from the back-end; rather, only the behavior of the system is being simulated regarding how it cost the CSB–UCC to retrieve the data. Also, the inter– arrival time (which represents the time between successive arrivals) is 10 seconds for 10000 requests. Thus, the arrival rate (which represents the number of jobs arrived per second) is 0.1 per second for every 10000 requests.
The results from the scalability test are graphed in Fig. 4.17 and the summary of values is presented in Table 4.22. The results reported are not single runs, but the averages of the repeated runs. Each experiment is repeated twenty (20) times and the standard deviation between the results within the repeated run is small (2.4 is recorded as the maximum standard deviation value throughout the entire experiment). In the first experiment, I focused on the centralized broker which is shown in the graph as 1 Broker. Starting from 10000 requests, the minimum throughput processed by the centralized broker is 2991 requests/second and the maximum throughput is approximately 32652 requests/second. The average (i.e., the mean) throughput is approximately 24632 requests/second. As shown in the graph, it is observed that the throughput value increases until the system almost stabilizes from 160000 requests. This range shows throughput value of around 32600 until a maximum capacity is reached at 250000 requests. The centralized broker then becomes non-responsive or crashes at this point. This shows that a centralize broker can handle up to about 30000 mobile users if those users are issuing approximately 8.3 requests/second. In the real world use case however, there can be more than this number of users or their request can be more than 8.3 requests/second.
So, the second set–up considers the performance of the system with two sub–proxies (i.e., 2 Brokers). For brevity, the reader is referred to Table 4.22 and Fig 4.17 to check the maximum, minimum, mean capacity, and maximum request capacity of each set–up in order to avoid the repetition of text. With 2 sub–proxies (2 Brokers), it is observed that the throughput value increases until the system almost stabilizes from 410000 requests. This range shows throughput value of about 60300 request/second until a maximum request capacity is reached at 520000 requests. The system then becomes non–responsive or crashes at this point. This performance is approximately double that of the centralized broker assuming we maintain the
Figure 4.17: The Scalability Test of the CSB-UCC Table 4.22: The Scalability Testing
1 Broker 2 Brokers 3 Brokers 4 Brokers 5 Brokers 6 Brokers Minimum Throughput (request/second) 2991 3022 3122 3002 3504 3507 Maximum Throughput (request/second) 32652 60502 93765 135001 166767 198902
Mean capacity (re- quest/second)
24632 44874 64539 80373 93083 112692
Maximum Request capac- ity (request/second)
250000 520000 790000 1000000 1000000 1000000
same mobile request rate at 8.3 requests/second.
With 3 Brokers, it is observed that the throughput value increases until the system almost stabilizes from 700000 requests. This range shows throughput value of about 90127 requests/second until a maximum request capacity is reached at 790000 requests. The system then becomes non–responsive or crashes at this point. This performance is approximately triple that of the centralized broker assuming we maintain the same mobile request rate at 8.3 requests/second.
When there are four sub–proxies (i.e., 4 Brokers), it is observed an increase in the throughput until the value stabilizes from 930000 requests. This range shows throughput value of about 130000 requests/second until a maximum request capacity is reached at 1000000 requests. The system then becomes non-responsive or crashes at this point.
In the set–up for 5 and 6 Brokers, the load generator could not issue more than a million requests. So, both experiments are limited to 1000000 requests capacity. I could not conclude this is the maximum number of users that 5 to 6 brokers can serve since the limitation is not from the system but from the experimental set–up. However, even though the maximum value is pegged at 1 million requests, it is observed that the throughput values have improved compared to the initial set–ups. With 5 Brokers, the throughput increases until it stabilizes from 960000 requests. This range shows throughput value of about 166600 requests/second. At the million request mark, the approximate throughput is 166767 requests/second. When the 6 Broker set–up is employed, a million requests shows approximate maximum request rate of 198902 requests/second. From the entire scalability test, it is observed that for a single broker, approximately 30000 users who own on average 8 mobile devices and can make 1 request/second from each device, to access services can be supported. There is an increase in the number of users by approximately 30000 for each additional sub–proxy that is added to the system. Another important observation from the graph is that, all the six brokers seem to have equal throughput at the beginning of the requests. The reason for this is that, the controller keeps sending the requests to one broker until that broker reaches saturation (where the broker starts sending busy message back to the controller). It is from this point that the controller will start sending the requests to another node. Hence, we realized that if the total request is say 170000, the throughput will be approximately the same for all the 6 brokers since at this number, regardless of the number of active brokers, only one broker will be working.