5. Probabilistic Algorithms
5.1 Coin-Tossing Algorithms
Probabilistic algorithms are able to toss coins. The control flow depends on the outcome of the coin tosses. Therefore, probabilistic algorithms exhibit random behavior.
Definition 5.1.Given an inputx, aprobabilistic (or randomized) algorithm
A may toss a coin a finite number of times during its computation of the outputy, and the next step may depend on the results of the preceding coin
136 5. Probabilistic Algorithms
tosses. The number of coin tosses may depend on the outcome of the previous ones, but it is bounded by some constant tx for a given input x. The coin tosses are independent and the coin is a fair one, i.e., each side appears with probability 1/2.
Examples. The encryption algorithms in Vernam’s one-time pad (Section 2.1), OAEP (Section 3.3.4) and ElGamal’s scheme (Section 3.5) include ran- dom choices, and thus are probabilistic, as well as the signing algorithms in PSS (Section 3.4.5), ElGamal’s scheme (Section 3.5.2) and the DSA (Section 3.5.3). Other examples of probabilistic algorithms are the algorithm for com- puting square roots inZ∗
p(see Algorithm A.61) and the probabilistic primality tests discussed in Appendix A.8. Many examples of probabilistic algorithms in various areas of application can be found, for example, in [MotRag95].
Remarks and Notations:
1. A formal definition of probabilistic algorithms can be given by the no- tion of probabilistic Turing machines ([LeeMooShaSha55]; [Rabin63]; [Santos69]; [Gill77]; [BalDiaGab95]).1 In a probabilistic Turing machine, the state transitions are determined by the input and the outcome of coin tosses. Probabilistic Turing machines should not be confused with non- deterministic machines. A non-deterministic Turing machine is “able to simply guess the solution to the given problem” and thus, in general, is not something that can be implemented in practice. A probabilistic ma- chine (or algorithm) is able to find the solution by use of its coin tosses, with some probability. Thus, it is something that can be implemented in practice.
Of course, we have to assume (and will assume in the following) that a random source of independent fair coin tosses is available. To implement such a source, the inherent randomness in physical phenomena can be exploited (see [MenOorVan96] and [Schneier96] for examples of sources which might be used in a computer).
To derive perfectly random bits from a natural source is a non-trivial task. The output bits may be biased (i.e., the probability that 1 is emitted is different from 1/2) or correlated (the probability of 1 depends on the
previously emitted bits). The outcomes of physical processes are often affected by previous outcomes and the circumstances that led to these outcomes. If the bits are independent, the problem of biased bits can be easily solved using the following method proposed by John von Neumann ([von Neumann63]): break the sequence of bits into pairs, discard pairs 00 and 11, and interpret 01 as 0 and 10 as 1 (the pairs 01 and 10 have the same probability). Handling a correlated bit source is more difficult. However, there are effective means of generating truly random sequences
1All algorithms are assumed to have a finite description (as a Turing machine)
which is independent of the size of the input. We do not consider non-uniform algorithms in this book.
of bits from a biased and correlated source. For example, Blum developed a method for a source which produces bits according to a known Markov chain ([Blum84]). Vazirani ([Vazirani85]) shows how almost independent, unbiased bits can be derived from two independent “slightly-random” sources. For a discussion of slightly random sources and their use in randomized algorithms, see [Papadimitriou94], for example.
2. The output y of a probabilistic algorithm A depends on the input x
and on the binary string r, which describes the outcome of the coin tosses. Usually, the coin tosses are considered as internal operations of the probabilistic algorithm. A second way to view a probabilistic algorithm
A is to consider the outcome of the coin tosses as an additional input, which is supplied by an external coin-tossing device. In this view, the model of a probabilistic algorithm is a deterministic machine. We call the corresponding deterministic algorithmADthedeterministic extensionof
A. It takes as inputs the original inputxand the outcomer of the coin tosses.
3. Given x, the output A(x) of a probabilistic algorithm A is not a single constant value, but a random variable. “A outputs y on input x” is a random event, and by prob(A(x) = y) we mean the probability of this event. More precisely, we have
prob(A(x) =y) := prob({r|AD(x, r) =y}).2
Here a question arises: what probability distribution of the coin tosses is meant? The question is easily answered if, as in our definition of prob- abilistic algorithms, the number of coin tosses is bounded by some con- stant tx for a given x. In this case, adding some dummy coin tosses, if necessary, we may assume that the number of coin tosses is exactly tx. Then the possible outcomesrof the coin tosses are the binary strings of lengthtx, and since the coin tosses are independent, we have the uniform distribution of {0,1}tx. The probability of an outcome r is 1/2tx, and
hence
prob(A(x) =y) =|{r|AD(x, r) =y}|
2tx .
It is sufficient for all our purposes to consider only probabilistic algo- rithms with a bounded number of coin tosses, for a givenx. In most parts of this book we consider algorithms whose running time is bounded by a functionf(|x|), where|x|is the size of the inputx. For these algorithms, the assumption is obviously true.
4. Given x, the probabilities prob(A(x) = y), y ∈ Y, define a probability distribution on the rangeY. We denote it bypA(x). The random variable
A(x) samplesY according to the distributionpA(x).
2 If the probability distribution is determined by the context, we often do not
specify the distribution explicitly and simply write prob(e) for the probability of an element or evente(see Appendix B.1).
138 5. Probabilistic Algorithms
5. The setting where a probabilistic algorithm A is executed may include further random events. Now, tossing a fair coin in A is assumed to be an independent random experiment. Therefore, the outcome of the coin tosses ofAon inputxis independent of all further random events in the given setting. In the following items, we apply this basic assumption. 6. Suppose that the input x ∈ X of a probabilistic algorithm A is ran-
domly generated. This means that a probability distributionpX is given for the domainX (e.g. the uniform distribution). We may consider the random experiment “Randomly choosex∈X according topX and com- pute y = A(x)”. If the outputs of A are in Y, then the experiment is modeled by a joint probability space (XY, pXY). The coin tosses ofA(x) are independent of the random choice of x. Thus, the probability that
x∈X is chosen and thaty=A(x) is
prob(x, A(x) =y) =pXY(x, y) =pX(x)·prob(A(x) =y). The probability prob(A(x) =y) is the conditional probability prob(y|x) of the outcomey, assuming the inputx.
7. Each execution ofA is a new independent random experiment: the coin tosses during one execution of A are independent of the coin tosses in other executions ofA. In particular, when executingAtwice, with inputs
xandx0, we have
prob(A(x) =y, A(x0) =y0) = prob(A(x) =y)·prob(A(x0) =y0).
If probabilistic algorithmsAandB are applied to inputsxandx0, then
the coin tosses ofA(x) andB(x0) are independent, unlessB(x0) is called
as a subroutine ofA(x) (such that the coin tosses ofB(x0) are contained
in the coin tosses ofA(x)), or vice versa:
prob(A(x) =y, B(x0) =y0) = prob(A(x) =y)·prob(B(x0) =y0).
8. LetAbe a probabilistic algorithm with inputs fromXand outputs inY. Leth:X −→Zbe a map yielding some propertyh(x) for the elements in
X (e.g. the least-significant bit ofx). LetB be a probabilistic algorithm which on input y ∈ Y outputs B(y) ∈ Z. Assume that a probability distributionpX is given onX.B might be an algorithm trying to invert
Aor at least trying to determine the propertyh(x) fromy:=A(x). We are interested in the random experiment “Randomly choosex, compute
y=A(x) andB(y), and check whetherB(y) =h(x)”. The random choice ofx, the coin tosses ofA(x) and the coin tosses ofB(y) are independent random experiments. Thus, the probability that x ∈ X is chosen, and thatA(x) =y andB(y) correctly computesh(x) is
prob(x, A(x) =y, B(y) =h(x))
9. LetpX be a probability distribution on the domainX of a probabilistic algorithmA with outputs inY. Randomly selecting anx∈X and com- puting A(x) is described by the joint probability spaceXY (see above). We can project to Y, (x, y)7→y, and calculate the probability distribu- tionpY onY: pY(y) := X x∈X pXY(x, y) = X x∈X pX(x)·prob(A(x) =y).
We call pY the image ofpX under A. pY is also the image of the joint distribution ofX and the coin tossesrunder the deterministic extension
ADofA:
pY(y) = prob({(x, r)|AD(x, r) =y})
= X
x∈X,r∈{0,1}tx:AD(x,r)=y
pX(x)·prob(r).
As in the deterministic case (see Appendix B.1, p. 330), we sometimes denote the image distribution by {A(x) :x←X}.
LetAbe a probabilistic algorithm with inputs fromX and outputs inY. ThenA(as a Turing machine) has a finite binary description. In particular, we can assume that both the domain X and the range Y are subsets of {0,1}∗. The time and space complexity of an algorithmA(corresponding to
the running time and memory requirements) are measured as functions of the binary length|x|of the inputx.
Definition 5.2.A probabilistic polynomial algorithm is a probabilistic algo- rithm A, such that the running time of A(x) is bounded by P(|x|), where
P ∈Z[X] is a polynomial (the same for all inputs x). The running time is measured as the number of steps in our model of algorithms, i.e., the number of steps of the probabilistic Turing machine. Tossing a coin is one step in this model.
Remark. The randomness in probabilistic algorithms is caused by random events in a very specific probability space, namely{0,1}tx with the uniform
distribution, and at first glance this might look restrictive. Actually, it is a very general model.
For example, suppose you want to control an algorithmAon inputxbyrx random events with probabilitiespx,1, . . . , px,rx (the deterministic extension AD takes as inputs x and one of the events).3 Assume that always one of the events occurs (i.e.,Prx
i=1px,i= 1) and that the probabilitiespx,i have a finite binary representationpx,i=
Ptx
j=1ax,i,j·2−j (ax,i,j∈ {0,1}). Further,
3 For example, think of an algorithmAthat attacks an encryption scheme. If all
possible plaintexts and their probability distribution are known, then Amight be based on the random choice of correctly distributed plaintexts.
140 5. Probabilistic Algorithms
assume thatrx, txand the probabilitiespx,iare computable by deterministic (polynomial) algorithms with input x. The last assumption is satisfied, for example, if the events and probabilities are the same for allx.
Then the random behavior ofA can be implemented by coin tosses, i.e.,
Acan be implemented as a probabilistic (polynomial) algorithm in the sense of Definitions 5.1 and 5.2. Namely, letS be the coin-tossing algorithm which on inputx:
(1) tx times tosses the coin and obtains a binary numberb:=btx−1. . . b1b0, with 0≤b <2tx, and
(2) returnsS(x) :=i, if 2txPi−1
j=1px,j ≤b <2tx Pi
j=1px,j.
The outputs ofS(x) are in{1, . . . , rx}and prob(S(x) =i) =px,i, for 1≤i≤
rx. The probabilistic (polynomial) algorithm S can be used to produce the random inputs for AD.