Creation of Workload Models

We describe two different ways of creating workload models. First we present a systematic manual approach for creating workload models from scratch. Second, we describe an automatic approach for inferring workload models from historical data. Using models, allows us to hide implementation de- tails and, instead, focus on relevant parts of describing user behavior. In performance testing, it is important that the load generated from workload models mimic the load generated by real users as closely as possible, other- wise it is not possible to draw any reliable conclusions from the test results [23]. Having automatic and systematic approaches for creating workload models ensure that our models stay close to real user behavior.

Systematic Creation of Workload Models

Our first proposal is a systematic approach for manually creating workload. Having a systematic approach is beneficial because it is repeatable and it also tells one how and where to obtain the necessary information. The first step in manually creating workload models is characterizing the workload of the system. Menasce and Almeida [56] state that the workload of a system can be defined as the set of all inputs the system receives from the environment during any given period of time. In modern systems, it is virtually impossible to construct probabilistic models that fully describe all possible interactions with a system, due to state space explosion. However, if possible, such models would still be difficult for humans to comprehend and certainly challenging to maintain. Hence, the workload of a system can be seen as a set of key performance scenarios. This idea is based on the fact that certain scenarios are more frequent than others and that some impact more on the performance of the system than other scenarios.

We start by analyzing the requirements, Service Level Agreements (SLAs), and the system specifications, respectively. By using these sources we iden- tify the inputs of the system with respect to types of transactions, transferred files, arrival rates, etc., following the generic guidelines discussed in [57]. Sec- ondly, we try to form an understanding of which are the different types of users and how they interact with the system. Finally, we identify what are the key performance scenarios for each user type that will impact most on the performance of the system. A user type can be seen as a set of users that share a common behavior and is characterized by the distribution and the types of actions it performs. Each identified user type is represented in the user profile and has a separate workload model that describes the prob- abilistic behavior. In addition, we extract information regarding the KPIs, such as the number of concurrent users the system should support, expected throughput, response times, expected resource utilization demands etc. for different actions under a given load. We would like to point out that this is a manual step in the process. The results of the workload characterization are aggregated in a workload model similar to the one in Figure 3.2

Automatic Creation of Workload Models

A second way of constructing a workload model is to use historical log data as a source of information [58]. We propose an automated approach that infers a set of workload models from web server log data. This approach is beneficial because it is less prone to errors, significantly reduces model creation time, and maps better to real user behavior compared to manual approaches. The starting point of this approach is a web server log provided by web servers such as Apache [59] or Microsoft Server [60]. A server log is a list of entries that describe requests for different types of resources. The entries in the log contain detail information about the requests, such as, IP- address, the time of request, the request method, the request resource, the status code, etc. Table 3.1 shows an example of a typical logging format.

IP-address User-Identifier User Id Date Requested Resource Status Size

87.153.57.43 example.com bob [20/Aug/2014:14:22:35 -0500] ”GET /browse HTTP/1.0” 200 855 136.242.54.78 example.com alice [20/Aug/2014:24:22:45 -0700] ”GET /browse HTTP/1.0” 200 855 87.153.57.43 example.com bob [20/Aug/2013:14:22:56 -0500] ”GET /basket/book 42/add HTTP/1.0” 200 685 136.242.54.78 example.com alice [20/Aug/2014:14:23:04 -0700] ”GET /basket/phone 6/add HTTP/1.0” 200 685 87.153.57.43 example.com bob [20/Aug/2013:14:23:58 -0500] ”GET /basket/book 42/delete HTTP/1.0” 200 936 136.242.54.78 example.com alice [21/Aug/2014:14:54:02 -0700] ”GET /basket/view.html HTTP/1.0” 200 1772

Table 3.1: Example of a typical web server format

By analyzing the server log it is possible to deduce the request pattern of individual users. In our approach, we analyze and process the server log in several steps in order to produce workload model. First, we parse the log file to extract the the information for the individual entries. During this

step, requests from autonomous machines, also referred to as bots, are ig- nored since they are considered as irrelevant candidates for key performance scenarios. Users are identified with their IP-address and requests made from the same users are kept is separate lists. Secondly, we split the list of request of each user into shorter list, called sessions, based on a predefined session time-out value. A session is a sequence of requests to the web server which represent the user activity in a certain time interval from the same user.

In the next step, we are trying to deduce user actions from a set of web requests. As stated earlier in this chapter, actions can be seen as abstract transactions or templates that fit many different requests. Actions can be quite similar in structure, yet, not identical to each other. For example, consider a normal web shop where users add products to the basket. Adding two different products to the basket will result in two different web requests even though the underlying user action is the same. To achieve this, we structure the request in a tree-like manner and, later, reducing the tree by grouping together nodes that share joint sub-nodes. Once the tree has been reduced to a minimum, every path leading to a leaf node is considered as an action.

We then classify users based on their request patterns using the K-means algorithm [61]. Users with a distinctly different access pattern are clustered in separate groups. This is done in order to obtain a separate workload models for user types with distinctly different behaviors. Before constructing workload models for each identified group of users, we filter out sessions with a low frequency according to a Pareto probability density function [62] by cutting off the tail beneath a certain threshold value. Sessions with a low frequency do not impact significantly on the system’s performance and can thus be neglected. Including all sessions would result in a workload model that is too cluttered and difficult to understand and maintain.

In a step-wise manner we then build a workload model where we overlap the remaining sessions of all users belonging to the same cluster. Session by session we gradually build a model, while reusing existing nodes in the model as much as possible. In order to calculate the probability and an average think time value for each edge we keep track of the number of times each edge has been reused.

Requirements Traceability in MBPeT

In our research, we also trace non-functional requirements across the model- based performance testing process. This allows us to compare the measured KPI values against target values set prior to testing. Target response time values are defined for individual action and monitored throughout the testing process. Whenever a target level is reached, the MBPeT tool reports on the current number of concurrent users and time of the breach. Figure 3.4 shows

a table where target response time values have been defined for individual actions. For example, for every action, an average and maximum threshold value is defined.

##### AVERAGE/MAX RESPONSE TIME THRESHOLD BREACH per METHOD CALL #####

Action Target Response Time NON­BIDDER_USER PASSIVE_USER AGGRESSIVE_USER Verdict

Average  (secs) Max  (secs) Average  users (secs) Max  users (secs) Average  users (secs) Max  users (secs) Average  users (secs) Max  users (secs)Pass/Fail GET_AUCTION(ID) 2.0 4.0 70 (251) 84 (299.0) 70 (251) 95 (341.0) 70 (250) 95 (341.0) Failed BROWSE() 4.0 8.0 84 (299) 97 (345.0) 84 (299) 113 (403.0) 84 (299) 113 (403.0) Failed GET_BIDS(ID) 3.0 6.0 84 (298) 112 (402.0) 83 (296) 112 (402.0) 96 (344) 112 (401.0) Failed BID(ID,PRICE,USERNAME,PASSWORD) 5.0 10 Passed Passed 97 (346) 113 (405.0) 112 (402) 135 (483.0) Failed SEARCH(STRING) 3.0 6 95 (341) 134 (479.0) 96 (342) 112 (402.0) 83 (296) 133 (476.0) Failed

Figure 3.4: Table showing traceability of response time values to ac- tions/method calls

After each test run, the measured average and maximum will be dis- played together with the target values. If any of the target values have been breached, the tool reports how long into the test run the threshold was breached and how many concurrent users the tool was running at that point. If the target value was not breached, the tool marks it with a pass. For ex- ample, in figure 3.4 we see that the target average response time value of 2 seconds for the get auction action for the aggressive user type was breached 250 seconds into the test run when running with 70 concurrent users. From this information we can conclude that; if the system must guarantee an av- erage response time of 2 seconds for the get auction action, given that a third of the users are of the type aggressive users, then the system cannot support more than 70 concurrent users. Besides just measuring response time values, the tool also monitors throughput, CPU, memory, disk, and network utilization.