Conditional Probability and Independence
3.6 Generating Functions
Even quite straightforward counting problems can lead to laborious and lengthy calcula-tions. These are often greatly simplified by using generating functions (introduced by de Moivre and Euler in the early eighteenth century). Later examples will show the utility of generating functions; in this section, we give a fairly bald list of basic definitions and properties, for ease of reference.
(1) Definition Given a collection of numbers (ai; i ≥ 0), the function ga(x)=∞
i=0
aixi
is called the generating function of (ai). (It is, of course, necessary that
aixiconverges somewhere if ga is defined as a function of x. If we regard ga as an element of a ring of
polynomials, such convergence is not necessary.)
(2) Definition Given (ai; i≥ 0), the function ha(x)=∞
i=0
aixi i !
is the exponential generating function of (ai).
(3) Definition Given a collection of functions ( fn(y); n≥ 0); the function g(x, y) =∞
n=0
xnfn(y)
is a bivariate generating function.
The following crucial result is a corollary of Taylor’s Theorem. We omit the proof.
(4) Theorem (Uniqueness) If for some x0 and x1we have ga(x)= gb(x)< ∞ for x0 < x < x1, then ai = bi for all i .
Generating functions help to tackle difference equations; the following result is typical.
(5) Theorem If (an; n≥ 0) satisfies a recurrence relation an+2+ 2ban+1+ can = dn; n≥ 0, then
ga(x)= x2gd(x)+ a0+ a1x+ 2ba0x 1+ 2bx + cx2 , with a corresponding result for higher-order equations.
3.6 Generating Functions 91 Proof To prove (5), multiply each side of the recurrence by xn+2and sum over n. The next four theorems are proved by rearranging the summation on the right-hand side in each case. You should do at least one as an exercise.
(6) Theorem (Convolution) If (an) can be written as a convolution of the sequences (bn) and (cn), so
an =
n i=0
cibn−i, n ≥ 0, then
ga(x)= gc(x)gb(x). (7) Theorem (Tails) If
bn =∞
i=1
an+i, n ≥ 0, then
gb(x)= ga(1)− ga(x) 1− x . (8) Theorem If
cn =
n i=0
ai, then
gc(x)= ga(x) 1− x.
(9) Theorem (Exponential Function) Let the function e(x) be defined by e(x)=∞
k=0
xk k!, then
e(x+ y) = e(x)e(y) = ex+y = exey. Finally, we have the celebrated
(10) Binomial Theorems For integral n, (1+ x)n =
n k=0
n k
xk, n ≥ 0 (11)
and
(1− x)−n=∞
k=0
n+ k − 1 k
xk n≥ 0, |x| < 1.
(12)
(13) Example Let us prove the binomial theorems.
Proof of (11) Considering the product (1+ x)n = (1 + x)(1 + x) . . . (1 + x), we see that a term xkis obtained by taking x from any choice of k of the brackets and taking 1 from the rest. Because there are (nk) ways of choosing k brackets, the term xk occurs (nk) times in the expansion. Because this is true for any k, the result follows.
Alternatively, you can prove (11) by induction on n, using the easily verified identity
n
Nowwe can prove (12) for arbitrary n in a number of ways; one possibility is by induction on n. Alternatively, we observe that a term xkis obtained in the product (1+ x + x2+ · · ·)n if we choose a set of k xs from the n brackets, where we may take any number of xs from each bracket. But by Theorem 3.3.4, this may be done in just (n+k−1k ) ways. The negative
binomial theorem (12) follows.
s
Here is a classic example.
(14) Example: The Coupon Collector’s Problem Each packet of an injurious product is equally likely to contain any one of n different types of coupon. If you buy r packets, what is the probability p(n, r) that you obtain at least one of each type of coupon?
Solution This famous problem can be approached in many different ways, as we see later, but the exponential generating function offers a particularly elegant answer.
First, recall from Theorem 3.2.3 that the probability of getting r =n
1ticoupons of n distinct types, where ti are of type i, is
n−rMr(t1, . . . , tn)= r ! nrt1!. . . tn!.
Then, p(n, r) is the sum of all such expressions in which ti ≥ 1 for all i (so we have at least one coupon of each type).
But nowexpanding (∞
t=1(s/n)t/t!)n by the multinomial theorem shows that the coef-ficient of sr in this expansion is just p(n, r)/r!. Hence,
and we have obtained the exponential generating function of the p(n, r).
s
Suppose that you are a more demanding collector who requires two complete sets of coupons, let p2(n, r) be the probability that r packets yield two complete sets of n coupons.
3.7 Techniques 93 Then exactly similar arguments showthat
exp
s n
− 1 − s n
n
= ∞
r=2n
sr
r !p2(n, r), and so on for more sets.
In conclusion, it is worth remarking that multivariate generating functions are often useful, although they will not appear much at this early stage. We give one example.
(15) Multinomial Theorem Recall the multinomial coefficients defined in Theorem 3.2.3,
Mn(n1, . . . , nr)= n!
r i=1
ni!
;
r i=1
ni = n.
We have
(x1+ x2+ · · · + xr)n =
Mn(n1, . . . , nr)x1n1x2n2. . . xrnr, where the sum is over all (n1, . . . , nr) such that
ni = n.
Proof This is immediate from the definition of Mn as the number of ways of permuting n symbols of which ni are of type i . In this case xi is of type i , of course. Corollary Setting x1= x2= · · · = xr = 1 gives
Mn(n1, . . . , nr)= rn.
3.7 Techniques
When evaluating any probability by counting, it is first essential to be clear what the sample space is, and exactly which outcomes are in the event of interest. Neglect of this obvious but essential step has led many a student into lengthy but nugatory calculations.
Second, it is even more important than usual to be flexible and imaginative in your approach. As the following examples show, a simple reinterpretation or reformulation can turn a hard problem into a trivial one. The main mechanical methods are:
(i) Using the theorems giving the numbers of ordered and unordered selections, recalling in particular that the number of ordered selections of r objects is the number of unordered selections multiplied by r !
(ii) Use of the inclusion–exclusion principle.
(iii) Setting up recurrence relations.
(iv) Use of generating functions.
Third, we remark that the implicit assumption of this chapter can be turned on its head.
That is, we have developed counting techniques to solve probability problems, but the solution of probability problems can just as well be used to prove combinatorial identities.
(1) Example Prove the following remarkable identity:
Hint: Consider an ant walking on a square lattice.
Solution An ant walks on the square lattice of points with nonnegative integer coor-dinates (i, j), i ≥ 0, j ≥ 0. It starts at (0, 0). If it is at (x, y), it proceeds next either to (x+ 1, y) or to (x, y + 1) with equal probability12. Therefore, at some transition (certainly less than 2n+ 1 transitions), it leaves the square
(0≤ x ≤ n, 0 ≤ y ≤ n).
However, Syoccurs if the ant has taken exactly y vertical steps before its (n+ 1)th hori-zontal step. There are (n+y+1y ) choices for the y vertical steps and each route to (n+ 1, y) has probability 2−(n+y+1). Hence,
P(Sy)=
Substituting (4) into (3) yields (2).
s
Finally, we remind the reader of our remark in 1.6 that many counting problems are traditionally formulated as “urn models.” They are used in this context for two reasons;
the first is utility. Using urns (instead of some realistic model) allows the student to see the probabilistic features without prejudice from false intuition. The second reason is traditional; urns have been used since at least the seventeenth century for lotteries and voting. (Indeed the French expression “aller aux urnes” means “to vote”.) It was therefore natural for probabilists to use them in constructing theoretical models of chance events.
A typical example arises when we look at the distribution of n accidents among days of the week, and seek the probability of at least one accident every day or the probability of accident-free days. The problem is equivalent to placing n balls in 7 urns, and seeking the number of ways in which the outcome of interest occurs.
The following result is central here.
(5) Theorem Let (x1, . . . , xn) be a collection of integers, such that x1+ · · · + xn = r.
The number of distinct ways of choosing them is:
(a)