4.5 Probability bounds
Note 10. We note that (4.19) follows directly from the Markov inequality Observe that
36. The differential entropy of a continuous random variable X with density f is
h(X) := E[−log f (X)] =
∞
−∞f(x)log
1
f(x)dx.
If X∼ uniform[0,2], find h(X). Repeat for X ∼ uniform[0,12] and for X ∼ N(m,σ2).
37. Let X have Student’s t density withνdegrees of freedom, as defined in Problem 20.
For n a positive integer less thanν/2, show that E[X2n] = νnΓ 2n+1 2 Γν−2n2 Γ1 2 Γν2 . 4.3: Transform methods
38. Let X have moment generating function MX(s) = eσ
2s2/2
. Use formula (4.8) to find E[X2].
39. Recall that the moment generating function of an N(0,1) random variable es2/2. Use
this fact to find the moment generating function of an N(m,σ2) random variable.
40. If X ∼ uniform(0,1), show that Y = ln(1/X) ∼ exp(1) by finding its moment gener- ating function for s< 1.
41. Find a closed-form expression for MX(s) if X ∼ Laplace(λ). Use your result to find
var(X).
42. Let X have the Pareto density f (x) = 2/x3for x≥ 1 and f (x) = 0 otherwise. For
what real values of s is MX(s) finite? Hint: It is not necessary to evaluate MX(s) to
answer the question.
43. Let Mp(s) denote the moment generating function of the gamma density gpdefined
in Problem 14. Show that
Mp(s) =
1
1− sMp−1(s), p > 1.
Remark. Since g1(x) is the exp(1) density, and M1(s) = 1/(1 − s) by direct calcula-
tion, it now follows that the moment generating function of an Erlang(m,1) random variable is 1/(1 − s)m.
44. Let X have the gamma density g
pgiven in Problem 14.
(a) For real s< 1, show that MX(s) = 1/(1 − s)p.
(b) The moments of X are given in Problem 29. Hence, from (4.10), we have for complex s, MX(s) = ∞
∑
n=0 sn n!· Γ(n + p) Γ(p) , |s| < 1.For complex s with |s| < 1, derive the Taylor series for 1/(1 − s)p and show that it is equal to the above series. Thus, MX(s) = 1/(1 − s)pfor all complex s
with|s| < 1. (This formula actually holds for all complex s with Res < 1; see Note 8.)
45. As shown in the preceding problem, the basic gamma density with parameter p, gp(x),
has moment generating function 1/(1−s)p. The more general gamma density defined
by gp,λ(x) :=λgp(λx) is given in Problem 15.
(a) Find the moment generating function and then the characteristic function of
gp,λ(x).
(b) Use the answer to (a) to find the moment generating function and the character- istic function of the Erlang density with parameters m andλ, gm,λ(x).
(c) Use the answer to (a) to find the moment generating function and the character- istic function of the chi-squared density with k degrees of freedom, gk/2,1/2(x).
46. Let X ∼ N(0,1), and put Y = X2. For real values of s< 1/2, show that MY(s) = 1 1− 2s 1/2 .
47. Let X ∼ N(m,1), and put Y = X2. For real values of s< 1/2, show that MY(s) =
esm2/(1−2s)
√ 1− 2s .
Remark. For m = 0, Y is said to be noncentral chi-squared with one degree of
freedom and noncentrality parameter m2. For m= 0, this reduces to the result of
the previous problem.
48. Let X have characteristic functionϕX(ν). If Y := aX +b for constants a and b, express
the characteristic function of Y in terms of a,b, andϕX.
49. Apply the Fourier inversion formula toϕX(ν) = e−λ|ν|to verify that this is the char-
acteristic function of a Cauchy(λ) random variable.
50. Use the following approach to find the characteristic function of the N(0,1) density
[62, pp. 138–139]. Let f(x) := e−x2/2/√2π. (a) Show that f(x) = −x f (x).
(b) Starting withϕX(ν) = ∞
−∞ejνxf(x)dx, computeϕX(ν). Then use part (a) to
show thatϕX(ν) = − j −∞∞ ejνxf(x)dx.
(c) Using integration by parts, show that this last integral is− jνϕX(ν).
(d) Show thatϕX(ν) = −νϕX(ν).
(e) Show that K(ν) :=ϕX(ν)eν
2/2
satisfies K(ν) = 0.
(f) Show that K(ν) = 1 for allν. (It then follows thatϕX(ν) = e−ν
2/2
.)
51. Use the method of Problem 50 to find the characteristic function of the gamma density gp(x) = xp−1e−x/Γ(p), x > 0. Hints: Show that (d/dx)xgp(x) = (p − x)gp(x). Use
integration by parts to show thatϕX(ν) = −(p/ν)ϕX(ν) + (1/ jν)ϕX(ν). Show that
K(ν) :=ϕX(ν)(1 − jν)psatisfies K(ν) = 0. 4.4: Expectation of multiple random variables
52. Let Z := X +Y, where X and Y are independent with X ∼ exp(1) and Y ∼ Laplace(1). Findcov(X,Z) and var(Z).
53. Find var(Z) for the random variable Z of Example 4.22.
54. Let X and Y be independent random variables with moment generating functions
MX(s) and MY(s). If Z := X −Y, show that MZ(s) = MX(s)MY(−s). Show that if
both X and Y are exp(λ), then Z ∼ Laplace(λ).
55. Let X1,...,Xnbe independent, and put Yn:= X1+ ··· + Xn.
(a) If Xi∼ N(mi,σi2), show that Yn∼ N(m,σ2), and identify m andσ2. In other
words, “The sum of independent Gaussian random variables is Gaussian.” (b) If Xi∼ Cauchy(λi), show that Yn∼ Cauchy(λ), and identifyλ. In other words,
(c) If Xiis a gamma random variable with parameters pi andλ (sameλ for all i),
show that Ynis gamma with parameters p andλ, and identify p. In other words,
“The sum of independent gamma random variables (with the same scale factor) is gamma (with the same scale factor).”
Remark. Note the following special cases of this result. If all the pi= 1, then
the Xiare exponential with parameterλ, and Ynis Erlang with parameters n and
λ. If p= 1/2 andλ = 1/2, then Xiis chi-squared with one degree of freedom,
and Ynis chi-squared with n degrees of freedom.
56. Let X1,...,Xr be i.i.d. gamma random variables with parameters p andλ. Let Y =
X1+ ··· + Xr. FindE[Yn].
57. Packet transmission times on a certain network link are i.i.d. with an exponential density of parameterλ. Suppose n packets are transmitted. Find the density of the time to transmit n packets.
58. The random number generator on a computer produces i.i.d. uniform(0,1) random variables X1,...,Xn. Find the probability density of
Y = ln n
∏
i=1 1 Xi .59. Let X1,...,Xnbe i.i.d. Cauchy(λ). Find the density of Y :=β1X1+···+βnXn, where
theβiare given positive constants.
60. Two particles arrive at a detector at random, independent positions X and Y lying on a straight line. The particles are resolvable if the absolute difference in their positions is greater than two. Find the probability that the two particles are not resolvable if X and Y are both Cauchy(1) random variables. Give a numerical answer.
61. Three independent pressure sensors produce output voltages U, V, and W, each exp(λ) random variables. The three voltages are summed and fed into an alarm that sounds if the sum is greater than x volts. Find the probability that the alarm sounds. 62. A certain electric power substation has n power lines. The line loads are independent
Cauchy(λ) random variables. The substation automatically shuts down if the total load is greater than. Find the probability of automatic shutdown.
63. The new outpost on Mars extracts water from the surrounding soil. There are 13 extractors. Each extractor produces water with a random efficiency that is uniformly distributed on [0,1]. The outpost operates normally if fewer than three extractors produce water with efficiency less than 0.25. If the efficiencies are independent, find the probability that the outpost operates normally.
64. The time to send an Internet packet is a chi-squared random variable T with one degree of freedom. The time to receive the acknowledgment A is also chi-squared with one degree of freedom. If T and A are independent, find the probability that the round trip time R := T + A is more than r.
65. In this problem we generalize the noncentral chi-squared density of Problem 47.
To distinguish these new densities from the original chi-squared densities defined in Problem 15, we refer to the original ones as central chi-squared densities. The non- central chi-squared density with k degrees of freedom and noncentrality parameter
λ2is defined byf ck,λ2(x) := ∞
∑
n=0 (λ2/2)ne−λ2/2 n! c2n+k(x), x > 0,where c2n+kdenotes the central chi-squared density with 2n+ k degrees of freedom.
Hence, ck,λ2(x) is a mixture density (Problem 12) with pn= (λ2/2)ne−λ
2/2
/n! being a Poisson(λ2/2) pmf.
(a) Show that 0∞ck,λ2(x)dx = 1.
(b) If X is a noncentral chi-squared random variable with k degrees of freedom and noncentrality parameterλ2, show that X has moment generating function
Mk,λ2(s) = exp[sλ
2/(1 − 2s)]
(1 − 2s)k/2 . Hint: Problem 45 may be helpful.
Remark. When k = 1, this agrees with Problem 47.
(c) Use part (b) to show that if X∼ ck,λ2, thenE[X] = k +λ2.
(d) Let X1,...,Xn be independent random variables with Xi∼ cki,λ2
i. Show that
Y := X1+ ··· + Xnhas the ck,λ2 density, and identify k andλ2.
Remark. By part (b), if each ki= 1, we could assume that each Xiis the square
of an N(λi,1) random variable.
(e) Show that
e−(x+λ2)/2
√ 2πx ·
eλ√x+ e−λ√x
2 = c1,λ2(x).
(Note that ifλ= 0, the left-hand side reduces to the central chi-squared density with one degree of freedom.) Hint: Use the power series eξ = ∑∞n=0ξn/n! for the two exponentials involving√x.
4.5:Probability bounds
66. Let X have the Pareto density f (x) = 2/x3for x≥ 1 and f (x) = 0 otherwise. For a≥ 1, compare P(X ≥ a) and the bound obtained via Markov inequality.
67. Let X be an exponential random variable with parameterλ= 1. Compute the Markov
inequality, the Chebyshev inequality, and the Chernoff bound to obtain bounds on P(X ≥ a) as a function of a. Also compute P(X ≥ a).
(a) For what values of a is the Markov inequality smaller than the Chebyshev in- equality?
(b) MATLAB. Plot the Markov bound, the Chebyshev bound, the Chernoff bound,
andP(X ≥ a) for 0 ≤ a ≤ 6 on the same graph. For what range of a is the Markov bound the smallest? the Chebyshev? Now use MATLAB command semilogy to draw the same four curves for 6≤ a ≤ 20. Which bound is the smallest?
Exam preparation
You may use the following suggestions to prepare a study sheet, including formulas men- tioned that you have trouble remembering. You may also want to ask your instructor for additional suggestions.
4.1. Densities and probabilities.Know how to compute probabilities involving a random variable with a density (4.1). A list of the more common densities can be found inside the back cover. Remember, density functions can never be negative and must integrate to one.
4.2. Expectation.LOTUS (4.3), especially for computing moments. The table inside the back cover contains moments of many of the more common densities.
4.3. Transform methods. Moment generating function definition (4.7) and moment for- mula (4.8). For continuous random variables, the mgf is essentially the Laplace trans- form of the density. Characteristic function definition (4.11) and moment formula (4.14). For continuous random variables, the density can be recovered with the in- verse Fourier transform (4.12). For integer-valued random variables, the pmf can be recovered with the formula for Fourier series coefficients (4.13). The table inside the back cover contains the mgf (or characteristic function) of many of the more common densities. Remember thatϕX(ν) = MX(s)|s= jν.
4.4. Expectation of multiple random variables. If X and Y are independent, then we haveE[h(X)k(Y)] = E[h(X)]E[k(Y)] for any functions h(x) and k(y). If X1,...,Xn
are independent random variables, then the moment generating function of the sum is the product of the moment generating functions, e.g., Example 4.23. If the Xiare
continuous random variables, then the density of their sum is the convolution of their densities, e.g., (4.16).
4.5. Probability bounds. The Markov inequality (2.18) and the Chebyshev inequality (2.21) were derived in Section 2.4. The Chernoff bound (4.20).
Work any review problems assigned by your instructor. If you finish them, re-work your homework assignments.