Tools specifically designed for cache benchmarking

Benchmarking a cache is not something which is done very often and by in- dividuals. There are quite a good number of web server benchmarking tools. Unlike web servers, for cache servers the choice is very limited.There are only a few number of open source tools exist. In the next section the most popular ones and their capabilities are shortly discussed. As mentioned previously the proxy caching topic was popular in early 2000s and most of today’s existing tools were developed then. Web polygraph is the only one which came with

2.6. CHALLENGES

updates afterwards.

WebJamma and proxysizer There are however a small number of tools which can read from a real trace and play back http access log files. WebJamma and proxysizer can generate workloads by reading existing traces and log files.

Winconsin is a tool that initially was made to benchmark the proxy servers. That is able to generate a very simple workload and supports the concept of server and client processes. However this tool was not updated afterwards and does not support many of new features in a web traffic trace. Among others HTTP/1.1 which is most commonly used protocol on the web traffic is not supported by the Winconsin.

Web polygraph is a powerful tool which claims to be able to generate a variety of workloads. It supports HTTP/1.1 and have got many features which enriches the load generated with almost all the needed characteristics. It simulates both clients and servers as well as generating the requests.

Chapter 3

Model and Methodology

In this chapter, the approach, test environment, benchmarking tools and basic cache server configuration parts will be discussed:

• Section3.1discusses the possible and proposed approaches to the problem. • Section 3.2 describes the test environment including hardware specifica-

tions and network configuration.

• Sections 3.3and 3.4describe the tools which were used for this work: – Web Polygraph

– blktrace – Seekwatcher – Custom scripts

• Sections 3.5through3.8describe the configuration specifications for Var- nish, Apache Traffic Server and Polygraph.

• Finally, Section 3.9 describes the various workloads designed for this ex- periment.

3.1

Approach

There are two dimensions that are considered in the proxy evaluation archi- tecture: the implementation environment and the source of workload. The possible workload sources divide into three categories: artificial, captured logs and current requests (the workload source space). Similarly, three main algo- rithms are available for the evaluations; simulated system and networks, real system/isolated networks and real system/real networks comprise the imple- mentation space. This is illustrated in Figure3.1

In an ideal test model, the following characteristics are desirable in order to get the best possible results:

3.1. APPROACH

• Flexibility of testing

• Observability of direct results

• Testing based on real traffic

• Testing performed on real systems

The first priority for this work is reproducibility. It is important to be able to reproduce tests as needed for both the Varnish server and Apache Traffic Server. In addition, the tests should be reproducible across runs in order to confirm the results. Furthermore, other people who might want to repeat the experiment should be able to do so.

Flexible testing is important because it provides the opportunity to produce scenarios that are interesting. Thus, by changing some characteristics of the configurations it is possible to create a variety of tests which can figure out how different variables affect the results.

Tests should be designed so that the results produced are direct measures of the behaviour and performance of the servers being tested.

Testing on real systems and real networks is also desirable because these envi- ronments give the most realistic results. Therefore, a scenario with real traffic on real systems would seem to be ideal. However, such a situation is not repro- ducible both because the state of a real scenario changes over time, and because real networks are inherently chaotic. A real system on an isolated network is a very good alternative. Since the main purpose of this work is to focus on performance of two products, an isolated network eliminates all the variations and unpredictability of a real network.

Doing the experiment with the real traffic again gives the most realistic results as it comprises real traffic patterns and the exact sizes of documents and their associated cost of retrieval. However, the main constraint in this case is again the reproducibility of the test. A good alternative which provides reproducibil- ity is using traffic generated from the captured logs of real servers running in production. Captured logs maintain all the characteristics of live traffic. Fur- thermore, such replayed traffic is reproducible. The problem, however, is the flexibility of the trace; captured logs cannot be tuned to assume whatever traffic characteristics become desirable as the experiment proceeds. The other issue with captured logs is the validity of objects, due to the fact that not all the ob- jects in the captured log trace are still valid or in existence at the time the trace is used for testing. An artificial workload does not have any of these problems.