Little’s Law &
Bottleneck Law
Dec
2011
IT professionals have shunned performance modelling considering it to be too complex and inapplicable to real life. A lot has to do with fear of mathematics as well. This tutorial provides the foundation of
performance modelling. Using elementary mathematics two
fundamental laws are discussed. Little’s Law is the most popular law in performance modelling and is applicable to all IT systems. The
bottleneck law is also very applicable and though used qualitatively, it is seldom applied quantitatively. This tutorial shows the applicability of both these laws to a variety of situations that are relevant to the IT industry.
Copyright (C) 2011 Rajesh Mansharamani
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled "GNU Free Documentation License". http://www.gnu.org/licenses/fdl.html
1
Little’s Law & Bottleneck Law
Two simple laws are introduced in this document, which have widespread applicability. These fundamental paradigms should suffice for a lot of quantitative analysis of IT systems, as will be shown through a set of examples provided and a set of exercises at the end of the document.
Before we get into the laws let us introduce the basic set of terms and notation that will be used to derive the laws. Thereafter we get in to Little‟s Law and then the Bottleneck Law. Little‟s Law provides a relationship between response time, number in the system and throughput. The Bottleneck Law shows where the system will bottleneck in terms of system throughput and using it we can derive useful response time and throughput bounds.
1. Definitions of Fundamental Terms
We define seven fundamental terms in this section, three are widely used in the software industry today, and the other four will be useful for the purpose of discussion. Response time, throughput, and number of concurrent users are popular terms used by software and IT systems professionals. The other four terms of interest to us are think time, service time, visit count, and demand.
By default all notation that we use will stand for averages, unless otherwise stated. Thus for example, R will be used to denote average response time.
1.1 System, Resource, Entity, and Work Conserving System
The fundamental terms to be defined are in the context of a system. As shown in Figure 1, a system is composed of a set of sub-systems, the most granular of which is a single resource. Entities flow through the system requesting services from various resources. For example, a data centre for a large bank is a system and a banking transaction such as a deposit is an entity. The banking transaction visits various system resources such as network, CPUs of the web, application, and database servers, as well as disks attached to the database server. In this context, a network switch is also a system, where the entities flowing through it are network packets. It is assumed that a resource can service only one request at any given instant of time. Thus, when a CPU is servicing a request at any instant of time, even if it is in time slicing mode, it is not servicing any other request at that point in time. Note that this discussion is only for one resource at an instant of time. A system is said to be work conserving if no work is generated within the system and no work gets destroyed within the system. For example, consider a bank, which provides for deposits and withdrawal transactions. Then work conserving means that a deposit or a withdrawal transaction is not generated unless there is a valid customer request, and once a transaction is accepted it is not denied by the bank.
2 Figure 1: System and Entity
1.2 Response Time
Response time of an entity is the time difference between entry and exit of the entity from the system. Thus, response time of a deposit transaction in the data centre is the time the transaction exits from the data centre network minus the time it enters in to the data centre network. However, if we consider end user response time, then we need to extend our system boundary to the end user terminals, so that the wide area network time as well as the time spent at the user terminal is counted.
The symbol R will be used to denote average response time. If we observe the system for a period of time T, and C entities leave the system during that time, with response times R1,...,RC then we can compute the average response time R as
R = (R1 + R2 + … + RC) / C
1.3 Throughput
Throughput of a system is its rate of processing entities. Thus number of banking transactions per day, number of web hits per second, number of I/Os per second, as well as network bandwidth in Kbps or Mbps, are all measures of throughput. The symbol X will be used to denote throughput. If we observe the system for a period of time T, and C entities leave the system during that time, then we have throughput as:
T
C
X
Throughput and response time are the two most important metrics for IT systems. A high throughput means more business transactions and a low response time means better end-user satisfaction.
1.4 Number in the System, Think Time, and Number of Concurrent Users
The number of entities being served in any system is a very useful measure for system capacity planning. We interchangeably use the terms „number in the system‟ and „number of entities in the system‟. This is a function of time, since at
3
different points in time the number in the system will vary. If the system is work conserving, then we have number in the system at any time t as:
N(t) = Number of Arrivals up to time t – Number of Departures up to time t
The symbol N will be used to denote average number in the system. If we observe the system for a period of time T, then we have average number in the system as1:
TN
t
dt
T
N
0)
(
1
A very widely used term in the IT industry is the „number of concurrent users‟. In this context, we are speaking of end-users. The user‟s activities will in turn impact the rate at which entities come in to the system. We will use the term „think time‟ to denote the time between receiving a response to a request and submitting the next request. The end user could be reviewing the response, or could be entering data in to the next request‟s screens, or could have stepped out and come back. Essentially, the user is still part of the system but has not submitted any entity for processing. In some literature think time is also referred to as sleep time. We will use the symbol Zto denote average think time.
As shown in Figure 2, the number of concurrent users in the system is essentially the number of end-users who are using the system; they could, for example, be either thinking or could have entities submitted for processing.
Figure 2: Concurrent Users
1.5 Service Time, Visit Count, and Demand
Service time is the time spent by a resource in servicing a single request by an entity. For example, a single banking transaction makes 5 requests to a web server with an average of 5ms CPU time per request, 2 requests to an application server with an average of 10ms CPU time per request, 4 requests to a database server
1
To find out average of a function, one needs to take a number of samples, add them up and divide the sum by the number of samples. For a function which changes rapidly with time, the number of samples needs to enough to cover all changes. Thus when the number of samples tends to infinity, the summation becomes an integral.
4
with an average of 20ms CPU time per request, and 10 requests to the disk subsystem with an average of 5ms disk service time per request. Note that the service times in this example are time spent in servicing the request, and they do not including queuing time or wait time at the resource, which forms part of response time. In other words, service time at a resource can be thought of as response time at the resource under idle conditions.
We use the symbol Sto denote average service time.
In the previous example we saw that a single transaction makes multiple visits to sub-systems and resources. The average number of visits to a resource is called the visit count of that entity at the resource. Note that visit count by definition is an average. Also note that visit count is a relative number. In the example above, one banking transaction makes 4 requests to the database server and 10 requests to the disk subsystem. Thus the visit count is 4 at the database server and 10 at the disk subsystem. This is relative to the banking transaction. At the disk subsystem the visit count relative to the database is 2.5 (10/4). Visit count can also be a fraction, which less than one. In the example above, if we have 8 CPUs at the database server, then the visit count per CPU is 4/8 = 0.5.
We use the symbol V to denote visit count.
Whether we make 4 requests to the database server with service time 20ms per request, or 1 request with service time 80ms, the total service demand at the database server remains the same, that is, 80ms. Thus the average demand at a resource is the product of average service time at that resource and the visit count at that resource.
The symbol D is used to denote average demand. Thus at each resource i in the system the average demand is:
i i
i
V
S
D
1.6 Summary of Notation
Table 1 summarizes the notation we will be using in the rest of this document.
Table 1: Notation
Symbol Description
D Average Demand
N Average Number in System
R Average Response Time
S Average Service Time
V Visit Count
X Throughput
5
2. Little’s Law
Little‟s Law, named after John D. C. Little, states that the average number in a work conserving system equals the product of the system throughput and the average response time. In this section, the proof for Little‟s Law will be worked out, and extended to end-user systems. The simplicity and high applicability of this law has resulted in it being widely used in performance modelling.
2.1 Little’s Law for any system
The technique used in the proof is operational analysis. Let us observe arrivals and departures in a system. As shown in Figure 3, let A(t) be the total number of arrivals up to time t, and D(t) be the total number of departures up to time t. Note that A(t) and D(t) are both non-decreasing functions. We also assume that each of them can be increased only in steps of 1 unit. In other words, the probability of two or more arrivals at exactly the same time and of two or more departures at exactly the same time is zero. This is not a limitation since we model two „simultaneous‟ arrivals as coming in at time t and time t+t, where t is an infinitesimally small interval of time.
Since the system is work conserving we cannot have more departures than arrivals, and hence D(t) ≤ A(t). By definition of number in the system at any point in time t, N(t) (see Section 1.4) we have
N(t) = A(t) – D(t)
Figure 3: Arrivals and Departures in a Work Conserving System
We are interested in determining the average number in the system, which from Section 1.4 is given by:
TN
t
dt
T
N
0)
(
1
The integral of N(t) is the area under the function, or the area between the functions A(t) and D(t), as shown in Figure 4.
6 Figure 4: Area Between Arrivals and Departures
The shaded area in Figure 4 is the sum of areas of the rectangles shown. Each rectangle is of height 1 (since we assumed that no two arrivals or no two departures happen at exactly the same instant of time). The width of any rectangle
i is the difference between the time of the ithdeparture, d
i and the time of the ith
arrival, ai. If the system has the same sequence of departures as the sequence of
arrivals, or in other words it is a first-come-first-serve (FCFS) system then di – ai is
the response of time job i, denoted by Ri. We will proceed with the FCFS
assumption and then prove the law can be extended for any other sequence of departures.
We now have the area of rectangle i as 1 x Ri = Ri. Therefore the average number
in the system is:
TN
t
dt
T
N
0)
(
1
=
C i iR
T
11
(1) Here C is the number of completions up to time T. In reality, equation (1) will hold for the limit T since we are not considering partial response time of any request being served at time T. Therefore, we need to assume that the interval of time T is „large enough‟ compared to individual response times.We can rewrite equation (1) by dividing and multiplying with C to get:
C
R
T
C
R
T
C i i C i i
1 11
(2) From Sections 1.3 and 1.2, we can see that the first term on the right hand side is the system throughput and the second term is the average response time. We therefore have:R
X
R
T
C i i
11
(3) Combining equations (1) and (3) we get:7
N = X R Little’s Law (4) To prove Little‟s Law we had assumed FCFS sequencing in the system. Let‟s now show that Little‟s Law applies for any type of sequencing. The key to Little‟s Law is equation (1) in which we have assumed that the shaded area in Figure 4 is the sum of the response times. While this is true for FCFS, why does it hold for any sequence of departures?
Even if the sequence of departures is not FCFS, there will be a one-to-one pairing between arrivals and departures. Figure 5 shows a sample mapping of arrivals to departures.
Figure 5: Sample mapping of arrivals to departures
Now let us renumber the departures as per their arrival sequence. If we order the entities in their sequence of arrivals, then let di’ denote the departure time of the ith
entity. Thus d1‟ is the departure time of the first entity, which by Figure 5, is
nothing but d2, as per the one is to one mapping of arrivals and departures. Figure
6, shows the departure times of Figure 5 renumbered as per the arrival sequence of entities.
Figure 6: Renumbering of departures
By this renumbering we therefore have response time of entity i as (di‟ – ai):
Ri = (di’ – ai) (5)
Coming back to the area of the shaded region in Figure 4, we see that the area is the sum of areas of rectangles each of height 1 and width (di – ai):
)
(
)
(
1 0 i C i i Ta
d
dt
t
N
(6)8
The summation on the right hand side can be expressed as:
)
(
)
(
1 ' 1 1 ' 1 1 1 i C i i C i i C i i C i i C i i i C i ia
d
a
d
a
d
a
d
(7) Combining equations (6), (7), and (5) we get
C i i i C i i i C i i TR
a
d
a
d
dt
t
N
1 1 ' 1 0)
(
)
(
)
(
(8)On account of equation (8), we have equation (1) holding for any sequence of departures, and thus Little‟s Law holds for any sequence of departures.
To summarize, we have just proved that average number in a system is the product of the system throughput and the average response time.
N = X R
2.2 Little’s Law for end-user systems
Consider this: if we have an IT system with 1,000 concurrent users, and average response time of 2 seconds, then can we say that by Little‟s Law the system throughput is 1000/2 = 500 requests/sec? This will not be correct since there may not be 1,000 requests in the system. To extend Little‟s Law to end-user systems, consider Figure 7, which shows a system whose response time is of interest, as well as the end-users of the system who have an average think time Z.
Figure 7: End-User System
In Figure 7, we consider an extended system shown by the dashed line, which includes the end-users. We have a total of N users in the extended system, each either thinking or waiting for a response. Thus, the total number in this system is N. The response time of this extended system is the sum of the think time and the response time of the system . Little‟s Law must hold for the extended system as well. We therefore have for end-user systems:
9
For the example we started this subsection with, if we have average think time as 18 seconds, then we will get the throughput as 1000/(2+18) = 50 requests/sec. Another simple way to prove Little‟s Law for end-user systems or closed systems is as follows. The cycle time for a single user is (R+Z). Therefore the throughput per user is 1/(R+Z) and therefore the throughput for N users is N/(R+Z).
In performance modelling when we look at a single resource or subsystem in isolation, we can view it as an open system and use N = X R, to get the average queue size or number in that subsystem or resource. This is also true for very large systems like google.com where it is impossible to estimate the number of users, but it is possible to estimate the rate of access, which means we can treat it as an open system. However, whenever we deal with a fixed number of users then the closed system model works best.
3. Bottleneck Law and Throughput and Response Time Bounds
Consider an end-user system as illustrated in Figure 8. An entity requesting services of the system visits several resources, with a certain visit count and average service time. The circles in the system denote resources, and the tuples shown next to the circles specify the visit count and average service time at the resources.
Figure 8: Visit Counts and Service Times
As defined in Section 1.5, the average demand at a resource is the product of the visit count and the service time. For the purpose of our analysis we can equate Figure 8 to Figure 9, which shows the system as a pipeline of resources each having service time equal to demand. In other words, instead of specifying that a resource
i is visited Vi times with an average service time of Si, we specify that the resource
is visited once with average demand of Di. For the purpose of the bounds derived in
10 Figure 9: Pipeline of demands
If we consider any pipelined system such as the one in Figure 9, the maximum throughput of the system cannot exceed the throughput at the slowest stage of the pipeline. In the example in Figure 9, the maximum throughput of the system is 1/5. Let the maximum average demand in the system, across all resources i, be denoted by Dmax:
}
{
max
max i iD
D
We therefore have the upper bound for system throughput as:
max
1
D
X
(11)We refer to this upper bound on throughput as the Bottleneck Law. In simpler terms you can go faster than the slowest stage in your system.
The upper bound holds, regardless of the system workload. When the system saturates this, the upper bound becomes an equality. By definition Dmax depends on
visit counts and service times. Dmax can be reduced by optimizing the software
design and implementation to reduce service times, or by using faster CPUs or disks to reduce service times, or by increasing the number of resources at a service centre to reduce the visit count per resource, or by changing the architecture of the system to reduce visit counts. For example, if database visit counts are high, one can either increase the number of CPUs or disks, or introduce caching at the application server in order to reduce the visit counts.
From Little‟s Law in equation (10) we get:
Z
X
N
11
Applying the upper bound (11) to equation (12) we get a lower bound on average response time:
Z
ND
R
max
(13)Bounds (11) and (13) become equalities upon system saturation (unless the system is not work conserving and thrashes after a certain load). We will see applications of these bounds in the next section.
The reader may refer to Lazowska et al. (see bibliography at the end of this chapter) for detailed analysis of bounds including upper bounds on response time and lower bounds on throughput, as well as bounds on balanced systems.
4. Examples on Little’s Law and Bottleneck Law
A number of real life examples on the usage of Little‟s Law and the bounds we have derived in this chapter are presented in other sections of this site. To sharpen the understanding of the law and the terminology introduced in this section we present several examples below and also encourage the reader to solve the exercises at the end of this document.
Example 1: A call centre for a credit card company wishes to plan on the number of employees it should have. The call centre receives 20,000 calls per day which is expected to grow to 30,000 per day over next 6 months. 75% of the calls occur during a peak 3 hour period. The average duration of a call is 5 minutes. The call centre would like its employees to be 70% utilized in attending to calls. What is the number of employees that they should plan for, over the next 6 months?
The peak throughput during next 6 months will be:
X = 75% x 30,000 / 3 hours = 7,500 calls per hour = 125 calls per minute If the average duration of a call is 5 minutes, then the cycle time at 70% utilization per employee is:
C = 5 minutes / 0.7 = 7.1 minutes
Therefore average number of employees required is given by Little‟s Law: N = X C = 125 x 7.1 = 888 employees
Example 2: XYZ.com has witnessed a surge in its web site traffic. It currently services a maximum throughput 5 million http hits per day across a farm of 10 web servers, each having 2 CPUs. They have ordered another 5 web servers of 2 CPUs each. What will be the http visit count per CPU? What is the throughput that can be serviced by the 15 servers, assuming full scalability at the web tier?
With 5 more servers coming in, the total number of servers becomes 15 and the total number of CPUs becomes 30. An http request can be serviced by any of the 30 CPUs and hence the http visit count per CPU is 1 / 30. Earlier the http visit
12
count per CPU was 1 / 20. Therefore the demand at the web server has come down in proportion to the visit count, that is, by 2/3rd. Therefore the throughput increases
from 5 million hits per day to 5 million x 3 / 2 (inversely proportional to demand) = 7.5 million hits per day.
Example 3: XYZ.com has now started using its site for revenue generation. With the upgraded set of 15 web servers it services 1 million business transactions per day. Each business transaction makes an average of 7.5 http calls and 10 database calls. Assume that 25% of the business transactions occur in peak one hour window and are uniformly spread within the peak hour. What is the maximum demand possible at the database server? If the database CPU average service time per database call is 5 milliseconds, then what is the minimum number of CPUs required at the database?
Peak throughput is:
X = 0.25 x 1 million / 1 hour = 0.25 million business transactions per hour = 70 business transactions per second To service 70 business transactions per second, the maximum demand has to be:
Dmax ≤ 1 / 70 = 14.3 milliseconds
Therefore demand at the database tier has to be less than 14.3 milliseconds since Dmax is the maximum demand across all resources. Using the definition of demand
we get:
Ddatabase = Vdatabase Sdatabase ≤ 14.3 milliseconds
Given that average service time per database call Sdatabase is 5 milliseconds we get
Vdatabase x 5 ≤ 14.3 Vdatabase ≤ 14.3 / 5 = 2.9
Now let‟s consider the visit count at the database tier per CPU. We have 10
database calls per business transaction. Therefore visit count per database CPU is: Vdatabase = 10 / (number of database CPUs) ≤ 2.9
Therefore we get:
Number of database CPUs 10/2.9
Therefore there must be a minimum of 4 CPUs at the database.
Example 4: In the previous example, the average service time at the database was 5 milliseconds. Assume that for the above throughput (1 million business transactions per day) the database CPU utilization during peak hours is 90%. By how much should the average service time be reduced to bring the database CPU utilization down to 70%?
As can be seen from the exercises section (Exercise 3), using Little‟s Law one can derive the following:
13
U = X S
Where U is the utilization of the resource, X is the throughput, and S the average service time at the resource. We are targeting the same throughput of 1 million business transactions per day but now wish to reduce the utilization. Using this relationship it is clear that average service time is directly proportional to utilization. Thus the new average service time per database call should be:
S = 0.7 / 0.9 x old service time = 0.7 / 0.9 x 5 = 3.9 milliseconds Useful Reading
Little‟s Law is named after its inventor and first appeared in “Little, J. D. C. A Proof of the Queuing Formula L = λ W. Operations Research, 9, 383-387 (1961)”. It has been widely used in a number of research papers in manufacturing, computer architecture, and networks.
The bounds for average response time and throughput, as well as further reading on queueing networks is provided in “Lazowska et al. Quantitative Systems Performance. Prentice Hall, 1984.” This book is widely referenced in the literature. Since it is out of print it has now been made available electronically from
http://www.cs.washington.edu/homes/lazowska/qsp/
Exercises
1. A request for proposal (RFP) states that the solution to be delivered must service 72,000 transactions per hour, with 2,000 concurrent users. What is the average cycle time per user?
2. A web server access log file (in extended format) shows a throughput of 15,000 hits per minute with an average response time of 200 milliseconds. What is the average number of active concurrent sessions at the web server?
3. Consider a subsystem with a number of identical resources, for example a CPU subsystem or a disk subsystem. Any request will undergo a service time and a wait time (if the resources are busy). Now focus only on the resource itself and not the queue for the resource. As shown in the figure below, draw the system boundary around the resources and apply Little‟s Law. Now prove that utilization of the resource U = X S, where X is the (sub)system throughput and S is the average service time at the resource. Note that for multiple resources at the subsystem U can be more than one. For example, if an 8 CPU server is 70% utilized then U is 5.6 (that is, 0.7 x 8).
14
4. You need to plan on network bandwidth for a large bank that has hundreds of branches all over the country. All branches connect to a central data centre over a wide area network (WAN). Branches come in flavours of small branches (5 users), medium branches (10 users), and large branches (20 users). Assume all users to be statistically identical. The bank‟s peak throughput at the data centre is 100 transactions per second and there are a total of 5,000 users across all branches. The bank wants an average of 1 second network delay between the branch and the data centre.
a. What is the throughput in transactions per second at small, medium, and large branches?
b. What is the (application level) bandwidth required for small, medium, and large branches if the network payload is 4KB per transaction? c. If the average response time target in the data centre is 1 second,
what is the think time per user?
Answers to exercises 1. 100 seconds 2. 50
3. By Little‟s Law average number in resource subsystem is throughput times average response time in subsystem. Average response time within the resource (outside of queueing) is average service time by definition. Average number in resource subsystem divided by number of resources is the percentage utilization of the subsystem. Hence U = X S.
4. a) Small branch 0.1 tps, medium branch 0.2 tps, large branch 0.4 tps b) 3.2Kbps, 6.4Kbps, and 12.8Kbps
c) 48 seconds
Throughput X Average Service Time S