ECE342 Course Notes

(1)

ECE342- Probability for

Electrical & Computer Engineers

C. Tellambura and M. Ardakani

Winter 2013

(2)

(3)

B 2009 Quizzes: Solutions 111 B.1 Quiz Number 1 . . . 111 B.2 Quiz Number 2 . . . 113 B.3 Quiz Number 3 . . . 114 B.4 Quiz Number 4 . . . 116 B.5 Quiz Number 5 . . . 118 B.6 Quiz Number 6 . . . 120 B.7 Quiz Number 7 . . . 122 B.8 Quiz Number 8 . . . 123 C 2010 Quizzes 125 C.1 Quiz Number 1 . . . 125 C.2 Quiz Number 2 . . . 126 C.3 Quiz Number 3 . . . 127 C.4 Quiz Number 4 . . . 128 C.5 Quiz Number 5 . . . 129 C.6 Quiz Number 6 . . . 130 C.7 Quiz Number 7 . . . 131 D 2010 Quizzes: Solutions 133 D.1 Quiz Number 1 . . . 133 D.2 Quiz Number 2 . . . 135 D.3 Quiz Number 3 . . . 136 D.4 Quiz Number 4 . . . 138 D.5 Quiz Number 5 . . . 139 D.6 Quiz Number 6 . . . 140 D.7 Quiz Number 7 . . . 141 E 2011 Quizzes 143 E.1 Quiz Number 1 . . . 143

E.2 Quiz Number 2 . . . 144

F 2011 Quizzes: Solutions 149 F.1 Quiz Number 1 . . . 149 F.2 Quiz Number 2 . . . 151 F.3 Quiz Number 3 . . . 153 F.4 Quiz Number 5 . . . 155 F.5 Quiz Number 6 . . . 157

(6)

(7)

Chapter 1 Basics of Probability Theory

Goals of EE387

• Introduce the basics of probability theory,

• Apply probability theory to solve engineering problems.

• Develop intuition into how the theory applies to practical situations.

1.1 Set theory

A set can be described by the tabular method or the description method.

Two special sets: (1) The universal set S and (2) The null set ϕ.

1.1.1 Basic Set Operations

|A|: cardinality of A.

A∪ B = {x|x ∈ A or x ∈ B}: union - Either A or B occurs or both occur. A∩ B = {x|x ∈ A and x ∈ B}: intersection - both A and B occur.

A− B = {x ∈ A and x /∈ B}: set diﬀerence Ac ={x | x ∈ S and x /∈ A}: complement of A.

n

∪

k=1

Ak = A1∪ A2∪ . . . ∪ An: Union of n≥ 2 events - one or more of Ak’s occur. n

∩

k=1

Ak = A1∩ A2∩ . . . ∩ An: Intersection of n≥ 2 events - all Ak’s occur

simulta-neously.

Deﬁnition 1.1: A and B are disjoint if A∩ B = ϕ.

Deﬁnition 1.2: A collection of events A1, A2, . . . , An (n ≥ 2) is mutually

(8)

1.1.2 Algebra of Sets

1. Union and intersection are commutative. 2. Union and intersection are distributive. 3. (A∪ B)c _{= A}c∩ Bc _{- De Morgan’s law.}

4. Duality Principle

1.2 Applying Set Theory to Probability

Deﬁnition 1.3: An experiment consists of a procedure and observations. Deﬁnition 1.4: An outcome is any possible observation of an experiment.

Deﬁnition 1.5: The sample space S of an experiment is the ﬁnest-grain,

mutually exclusive, collectively exhaustive set of all possible outcomes.

Deﬁnition 1.6: An event is a set of outcomes of an experiment.

Deﬁnition 1.7: A set of mutually exclusive sets (events) whose union equals

the sample space is an event space of S. Mathematically, Bi∩ Bj = ϕ for all

i̸= j and B1∪ B2∪ . . . ∪ Bn= S.

Theorem 1.1: For an event space B ={B1, B2,· · · , Bn} and any event A ⊂ S,

let Ci = A∩ Bi, i = 1, 2,· · · , n. For i ̸= j, the events Ci and Cj are mutually

exclusive, i.e., Ci∩ Cj = ϕ, and A = n

∪

i=1

Ci.

1.3 Probability Axioms

Deﬁnition 1.8: Axioms of Probability: A probability measure P [·] is a

function that maps events in S to real numbers such that:

Axiom 1. For any event A, P [A] ≥ 0. Axiom 2. P [S] = 1.

Axiom 3. For any countable collection A1, A2,· · · of mutually exclusive events P [A1∪ A2∪ · · · ] = P [A1] + P [A2] +· · ·

(9)

1.4 Some Consequences of Probability Axioms 3

Theorem 1.2: If A = A1 ∪ A2 ∪ · · · ∪ Am and Ai∩ Aj = ϕ for i ̸= j, then

P [A] =

m

∑

i=1

P [Ai].

Theorem 1.3: The probability of an event B = {s1, s2,· · · , sm} is the sum of

the probabilities of the outcomes in the event, i.e., P [B] =

m

∑

i=1

P [{si}].

1.4 Some Consequences of Probability Axioms

Theorem 1.4: The probability measure P [·] satisﬁes 1. P [ϕ] = 0. 2. P [Ac_{] = 1}− P [A].

3. For any A and B (not necessarily disjoint), P [A∪B] = P [A]+P [B]−P [A∩B]. 4. If A ⊂ B, then P [A] ≤ P [B].

Theorem 1.5:

For any event A and event space B = {B1, B2,· · · , Bm} ,

P [A] = m ∑ i=1 P [A∩ Bi].

1.5 Conditional probability

The probability in Section 1.3 is also called a priori probability. If an event has happened, this information can be used to update the a priori probability.

Deﬁnition 1.9: The conditional probability of event A given B is

P [A|B] = P [A∩ B] P [B] .

To calculate P [A|B], ﬁnd P [A ∩ B] and P [B] ﬁrst.

Theorem 1.6 (Law of total probability): For an event space

{B1, B2,· · · , Bm} with P [Bi] > 0 for all i,

P [A] =

m

∑

i=1

P [A|Bi]P [Bi].

Theorem 1.7 (Bayes’ Theorem): P [B|A] = P [A|B]P [B]

(10)

Theorem 1.8 (Bayes’ Theorem- Expanded Version): P [Bi|A] = P [A|Bi]P [Bi] ∑m i=1P [A|Bi]P [Bi] .

1.6 Independence

Deﬁnition 1.10: Events A and B are independent if and only if P [A∩ B] =

P [A]P [B].

Relationship with conditional probability: P [A|B] = P [A], P [B|A] = P [B]

when A and B are independent.

Deﬁnition 1.11: Events A, B and C are independent if and only if

P [A∩ B] = P [A]P [B] P [B∩ C] = P [B]P [C] P [A∩ C] = P [A]P [C] P [A∩ B ∩ C] = P [A]P [B]P [C].

1.7 Sequential experiments and tree diagrams

Many experiments consist of a sequence of trials (subexperiments). Such experi-ments can be visualized as multiple stage experiexperi-ments.

Such experiments can be conveniently represented by tree diagrams.

The law of total probability is used with tree diagrams to compute event proba-bilities of these experiments.

1.8 Counting Methods

Deﬁnition 1.12: If task A can be done in n ways and B in k way, then A and

B can be done in nk ways.

Deﬁnition 1.13: If task A can be done in n ways and B in k way, then either

A or B can be done in n + k ways.

Here are some important cases:

• The number of ways to choose k objects out of n distinguishable objects (with replacement and with ordering) is nk_.

(11)

1.9 Reliability Problems 5

• The number of ways to choose k objects out of n distinguishable objects (without replacement and with ordering) is n(n− 1) · · · (n − k + 1).

• The number of ways to choose k objects out of n distinguishable objects (without replacement and without ordering) is(n_k)= n!

k!(n− k)!.

• Number of permutations on n objects out of which n1 are alike, n2 are alike, . . ., nR are alike:

n! n1!n2!· · · nR!

.

1.9 Reliability Problems

For n independent systems in series: P [W ] =

n

∏

i=1

P [Wi].

For n independent systems in parallel: P [W ] = 1−

n

∏

i=1

(1− P [Wi]).

1.10 Illustrated Problems

1. True or False. Explain your answer in one line.

a) If A = {x2_{|0 < x < 2, x ∈ R} and B = {2x|0 < x < 2, x ∈ R} then} A = B.

b) If A⊂ B then A ∪ B = A

c) If A⊂ B and B ⊂ C then A ⊂ C d) For any A, B and C, A∩ B ⊂ A ∪ C

e) There exist a set A for which (A∩ ∅c₎c∩S = A (S is the universal set).

f) For a sample space S and two events A and C, deﬁne B1 = A∩C,B2 = Ac∩ C, B

3 = A∩ Cc and B4 = Ac ∩ Cc. Then {B1, B2, B3, B4}is an

event space.

2. Using the algebra of sets, prove

a) A∩ (B − C) = (A ∩ B) − (A ∩ C), b) A− (A ∩ B) = A − B.

3. Sketch A− B for a) A⊂ B, b) B ⊂ A,

(12)

4. Consider the following subsets of S = {1, 2, 3, 4, 5, 6}: R1 ={1, 2, 5}, R2 = {3, 4, 5, 6}, R3 ={2, 4, 6}, R4 ={1, 3, 6}, R5 ={1, 3, 5}. Find: a) R1∪ R2, b) R4∩ R5, c) Rc₅, d) (R1∪ R2)∩ R3, e) Rc 1∪ (R4∩ R5), f) (R1∩ (R2∪ R3))c, g) ((R1∪ Rc2)∩ (R4∪ R5c))c

h) Write down a suitable event space.

5. Express the following sets in R as a single interval: a) ((−∞, 1) ∪ (4, ∞))c_,

b) [0, 1]∩ [0.5, 2], c) [−1, 0] ∪ [0, 1].

6. By drawing a suitable Venn diagram, convince yourself of the following: a) A∩ (A ∪ B) = A,

b) A∪ (A ∩ B) = A.

7. Three telephone lines are monitored. At a given time, each telephone line can be in one of the following three modes: (1) Voice Mode, i.e., the line is busy and someone is speaking (2) Data Mode, i.e., the line is busy with a modem or fax signal and (3) Inactive Mode, i.e., the line is not busy. We show these three modes with V, D and I respectively. For example if the ﬁrst and second lines are in Data Mode and the third line is in Inactive Mode, the observation is DDI.

a) Write the elements of the event A= {at least two Voice Modes} b) Write the elements of B= {number of Data Modes > 1+ number of

Voice modes}

8. The data packets that arrive at an Internet switch are buffered to be pro-cessed. When the buffer is full, the arrived packet is dropped and the trans-mission must be repeated. To study this system, at the arrival time of any new packet, we observe the number of packets that are already stored in the buffer. Assuming that the switch can buffer a maximum of 5 packets, the used buffer at any given time is 0, 1, 2, 3, 4 or 5 packets. Thus the sample space for this experiment is S ={0, 1, 2, 3, 4, 5}.

(13)

1.10 Illustrated Problems 7

Used buﬀer Number of times observed

0 112 1 119 2 131 3 85 4 43 5 10

The relative frequency of an event A is deﬁned as nA

n , where nAis the number

of timesA occurs and n is the total number of observations.

a) Consider the following three exclusively mutual events:A = {0, 1, 2},

B ={3, 4}, C = {5}. Find the relative frequency of these events.

b) Show that the relative frequency of A∪ B ∪ C is equal to the sum of the relative frequencies of A, B and C.

9. Consider an elevator in a building with four stories, 1-4, with 1 being the ground floor. Three people enter the elevator on floor 1 and push buttons for their destination floors. Let the outcomes be the possible stopping patterns for all passengers to leave the elevator on the way up. For example, 2-2-4 means the elevator stops on floors 2 and 4. Therefore,

2-2-4 is an outcome in S.

a) List the sample space, S, with its elements (outcomes).

b) Consider all outcomes equally likely. What is the probability of each outcome?

c) Let E = {stops only on even ﬂoors} and T = {stops only twice}. Find P[E] and P[T ].

d) Find P [E∩ T ] e) Find P [E∪ T ]

f) Is P [EU T ] = P [E] + P T ]? Does this contradict the third axiom of probability?

10. This problem requires the use of event spaces. Consider a random exper-iment and four events A, B, C, and D such that A and B form an event space and also C and D form an event space. Furthermore, P [A∩ C] = 0.3 and P [B∩ D] = 0.25.

a) Find P [A∪ C].

b) If P [D] = 0.58, ﬁnd P [A]. 11. Prove the following inequalities:

(14)

a) P [A∪ B] ≤ P [A] + P [B]. b) P [A∩ B] ≥ P [A] + P [B] − 1.

12. This problem requires the law of total probability and conditional

probability. A study on relation between the family size and the number

of cars reveals the following probabilities. Number of Cars

Family size 0 1 2 More than 2

S: Small (2 or less) 0.04 0.14 0.02 0.00 M: Medium (3, 4 or 5) 0.02 0.33 0.23 0.02 L: Large(more than 5) 0.01 0.03 0.13 0.03 Answer the following questions:

a) What is the probability of a random family having less than 2 cars? b) Given that a family has more than 2 cars, what is the probability that

this family be large?

c) Given that a family has less than 2 cars, what is the probability that this family be large?

d) Given that the family size is not medium, what is the probably of having one car?

13. A communication channel model is shown Fig. 1.1. The input is either 0 or 1, and the output is 0, 1 or X, where X represents a bit that is lost and not arrived at the channel output. Also, due to noise and other imperfections, the channel may transmit a bit in error. When Input = 0, the correct output (Output = 0) occurs with a probability of 0.8, the incorrect output (Output = 1) occurs with a probability of 0.1, and the bit is lost (Output = X) with a probability of 0.1. When Input = 1, the correct output (Output = 1) occurs with a probability of 0.7, the wrong output (Output = 0) occurs with a probability of 0.2, and the bit is lost (Output = X) with a probability of 0.1. Assume that the inputs 0 and 1 are equally likely (i.e. P [0] = P [1]).

a) If Output = 1, what is the probability of Input = 1?

b) If the output is X, what is the probability of Input = 1, and what is the probability of Input = 0?

c) Repeat part a), but this time assume that the inputs are not equally likely and P [0] = 3P [1].

14. This problem requires Bayes’ theorem. Considering all the other evidences Sherlock was 60% certain that Jack is the criminal. This morning, he found

(15)

0

1

1 X

0 Input

Output

Figure 1.1: Communication Channel

another piece of evidence proving that the criminal is left handed. Dr. Watson just called and informed Sherlock that on average 20% of people are left handed and that Jack is indeed left handed. How certain of the guilt of Jack should Sherlock be after receiving this call?

15. This problem requires Bayes’ theorem. Two urns A and B each have 10 balls. Urn A has 3 green, 2 red and 5 white balls and Urn B has 1 green, 6 red and 3 white balls. One urn is chosen at (equally likely) and one ball is drawn from it (balls are also chosen equally likely)

a) What is the probability that this ball is red?

b) Given that the drawn ball is red, what the probability that Urn A was selected?

c) Suppose the drawn ball is green. Now we return this green ball to the other urn and draw a ball from it (from the urn that received the green ball). What is the probability that this ball is red?

16. Two urns A with 1 blue and 6 red balls and B with 6 blue and 1 red balls are present. Flip a coin. If the outcomes is H, put one random ball from A in B, and if the outcome is T , put one random ball from B in A. Now draw a ball from A. If blue, you win. If not, draw a ball from B, if blue you win, if red, you lose. What is the probability of wining this game?

17. Two coins are in an urn. One is fair with P [H] = P [T ] = 0.5, and one is biased with P [H] = 0.25 and P [T ] = 0.75. One coin is chosen at random (equally likely) and is tossed three times.

a) Given that the biased coin is selected what is the probability of T T T ? b) Given that the biased coin is selected and that the outcome of the ﬁrst

(16)

A C B

Figure 1.2: for question 20.

c) This time, assume that we do not know which coin is selected. We observe that the ﬁrst three outcomes are T T T . What is the probability that the next outcome is T ?

d) Deﬁne two events E1: the outcomes of the ﬁrst three tosses are T T T ;

E2: the forth toss is T . Are E1 and E2 independent?

e) Given that the biased coin is selected, are E1 and E2 independent? 18. Answer the following questions about rearranging the letters of the word

“toronto”

a) How many diﬀerent orders are there?

b) In how many of them does ‘r’ appear before n?

c) In how many of them the middle letter is a consonant? d) How many do not have any pair of consecutive ‘o’s?

19. Consider a class of 14 girls and 16 boys. Also two of the girls are sisters. A team of 8 players are selected from this class at random.

a) What is the probability that the team consists of 4 girls and 4 boys? b) What is the probability that the team be uni-gender (all boys or all

girls)?

c) What is the probability that the number of girls be greater than the number of boys?

d) What is the probability that both sisters are in the team?

20. In the network (Fig. 1.2), a data packet is sent from A to B. In each step, the packet can be sent one block either to the right or up. Thus, a total of 9 steps are required to reach B.

a) How many paths are there from A to B?

b) If one of these paths are chosen randomly (equally likely), what is the probability that it pass through C?

(17)

1.11 Solutions for the Illustrated Problems 11 R 1 R R R a b Figure 1.3: Question 22.

21. A binary communication system transmits a signal X that is either a + 2 voltage signal or a− 2 voltage signal. These voltage signals are equally likely. A malicious channel reduces the magnitude of the received signal by the number of heads it counts in two tosses of a coin. Let Y be the resulting signal.

a) Describe the sample space in terms of input-output pairs.

b) Find the set of outcomes corresponding to the event ‘transmitted signal was deﬁnitely +2’.

c) Describe in words the event corresponding to the outcome Y = 0. d) Use a tree diagram to ﬁnd the set of possible input-output pairs.

e) Find the probabilities of the input-output pair. f) Find the probabilities of the output values.

g) Find the probability that the input was X = +2 given that Y = k for all possible values of k.

22. In a communication system the signal sent from point a to point b arrives along two paths in parallel (Fig. 1.3). Over each path the signal passes through two repeaters in series. Each repeater in Path 1 has a 0.05 prob-ability of failing (because of an open circuit). This probprob-ability is 0.08 for each repeater on Path 2. All repeaters fail independently of each other.

a) Find the probability that the signal will not arrive at point b.

1.11 Solutions for the Illustrated Problems

1. a) True. They both contain all real numbers between 0 and 4. b) False. A∪ B = B

c) True. ∀x ∈ A ⇒ x ∈ B ⇒ x ∈ C, therefore: A ⊂ C. d) True. Because (A∩ B) ⊂ A and A ⊂ A ∪ C.

e) False. (A∩ ϕc₎c

= (A∩ S)c = (A)c = Ac_{. There is no set A such}

(18)

f) True. Bis are mutually exclusive and collectively exhaustive.

2. a) Starting with the left hand side we have:

A∩ (B − C) = A ∩ (B ∩ Cc) = (A∩ B) ∩ Cc = (A∩ B) ∩ Cc = (A∩ B) − C.

For the right hand side we have:

(A∩ B) − (A ∩ C) = (A ∩ B) ∩ (A ∩ C)c = (A∩ B) ∩ (Ac ∪ Cc) = (A∩ B ∩ Ac)∪ (A ∩ B ∩ Cc)

We also know that (A∩ B ∩ Ac_{) = ((A}∩ Ac₎∩ B) = ϕ and

(A∩ B ∩ Cc) = ((A∩ B) ∩ Cc) = (A∩ B) − C.

As a result we have: (A∩ B) − (A ∩ C) = ϕ ∪ ((A ∩ B) − C) = (A∩ B) − C.

Then both sides are equal to (A∩ B) − C, and therefore the equality holds. b) A− (A ∩ B) = A ∩ (A ∩ B)c _{= A}_{∩ (A}c_{∪ B}c_{) = (A}_{∩ A}c₎_{∪ (A ∩ B}c₎ We know that A∩ Ac _{= ϕ.} Thus we have: A− (A ∩ B) = ϕ ∪ (A ∩ Bc) = A∩ Bc = A− B 3. a) null set b) A (A – B) B c) B A (A – B) S 4. a) R1∪ R2 ={1, 2, 3, 4, 5, 6} b) R4∩ R5 ={1, 3}

(19)

1.11 Solutions for the Illustrated Problems 13 c) Rc₅ ={2, 4, 6} d) (R1∪ R2)∩ R3 ={2, 4, 6} e) Rc 1∪ (R4∩ R5) ={1, 3, 4, 6} f) (R1∩ (R2∪ R3))c ={1, 3, 4, 6} g) ((R1∪ Rc2)∩ (R4∪ R5c))c ={3, 4, 5, 6}

h) One solution is {1,2,3} and {4,5,6} which partition S to two disjoint sets.

5. a) ((−∞, 1) ∪ (4, ∞))c _{= [1, 4]}

b) [0, 1]∩ [0.5, 2] = [0.5, 1] c) [−1, 0] ∪ [0, 1] = [−1, 1] 6. Try drawing Venn diagrams

7. A = {V V I, V V D, V V V, V IV, V DV, IV V, DV V }

B ={DDD, DDI, DID, IDD}

8. a) nA n = 112+119+131 500 = 0.724 nB n = 85+43 500 = 0.256 nC n = 10 500 = 0.02 b) nA∪B∪C n = 500 500 = 1 nA n + nB n + nC n = 0.724 + 0.256 + 0.02 = 1 = nA∪B∪C n 9. a) S ={2 − 2 − 2, 2 − 2 − 3, 2 − 2 − 4, 2 − 3 − 3, 2 − 3 − 4, 2 − 4 − 4, 3− 3 − 3, 3 − 3 − 4, 3 − 4 − 4, 4 − 4 − 4}

b) There are 10 elements in S, thus the probability of each outcome is 1/10. To be mathematically rigorous, one can deﬁne 10 mutually exclu-sive outcomes: E1 ={2−2−2}, E2 ={2−2−3}, . . ., E10 ={4−4−4}.

These outcomes are also collectively exhaustive.

Thus, using the second and the third axioms of probability, P [E1] + P [E2] + ...P [E10] = P [S] = 1.

Now, since these outcomes are equally likely, each has P [Ei] = 1/10.

c) E ={2 − 2 − 2, 2 − 2 − 4, 2 − 4 − 4, 4 − 4 − 4}, T ={2 − 2 − 3, 2 − 2 − 4, 2 − 3 − 3, 2 − 4 − 4, 3 − 3 − 4, 3 − 4 − 4} Thus, P [E] = 4/10, P [T ] = 6/10 d, e) E∩ T = {2 − 2 − 4, 2 − 4 − 4} E∪ T = {2 − 2 − 2, 2 − 2 − 3, 2 − 2 − 4, 2 − 3 − 3, 2 − 4 − 4, 3 − 3 − 4, 3− 4 − 4, 4 − 4 − 4} Thus, P [E∩ T ] = 2/10, P [E ∪ T ] = 8/10.

(20)

f) It can be seen that P [E∪ T ] ̸= P [E] + P [T ]. This does not contradicts the third axiom, because the third axiom on only for mutually exclusive (in this case, disjoint) events. E and T are not disjoint.

10. A = Bc and C = Dc a) P [B∩ D] = P [Cc∩ Ac] = P [(A∪ C)c] = 1− P [A ∪ C] ⇒ P [A ∪ C] = 0.75 b) P [A∪ C] = P [A] + P [C] − P [A ∩ C] ⇒ P [A] = P [A ∪ C] − P [C] + P [A ∩ C] P [C] = 1− P [D] = 0.42 ⇒ P [A] = 0.75 − 0.42 + 0.3 = 0.63 11. a) P [A∪ B] = P [A] + P [B] − P [A ∩ B] P [A∩ B] ≥ 0 } ⇒ P [A ∪ B] ≤ P [A]+P [B]

Notice that from a) it can easily be concluded that

P [A∪ B ∪ C ∪ · · ·] ≤ P [A] + P [B] + P [C] + · · · b) P [A∪ B] = P [A] + P [B] − P [A ∩ B] P [A∪ B] ≤ 1 } ⇒ P [A]+P [B]−P [A ∩ B] ≤ 1 ⇒ P [A ∩ B] ≥ P [A] + P [B] − 1

12. a) We deﬁne A to be the event that a random family has less than two cars and N to be number of cars.

P [A∩ S] = P [N = 0 ∩ S] + P [N = 1 ∩ S] = 0.04 + 0.14 = 0.18 P [A∩ M] = P [N = 0 ∩ M] + P [N = 1 ∩ M] = 0.02 + 0.33 = 0.35 P [A∩ L] = P [N = 0 ∩ L] + P [N = 1 ∩ L] = 0.01 + 0.03 = 0.04 P [A] = P [A∩ S] + P [A ∩ M] + P [A ∩ L] = 0.18 + 0.35 + 0.04 = 0.57 b) P [ L| N > 2] = P [L_{P [N >2]}∩(N>2)] = _{P [L}_{∩(N>2)]+P [M∩N>2]+P [S∩(N>2)]}P [L∩(N>2)] ⇒ P [L| N > 2] = 0.03 0.03+0.02+0 = 0.6 c) P [L|N < 2] = P [L_{P [N <2]}∩(N<2)] = _{P [L}_{∩(N<2)]+P [M∩N<2]+P [S∩(N<2)]}P [L∩(N<2)] ⇒ P [L| N < 2] = 0.03+0.01 (0.03+0.01)+(0.33+0.02)+(0.14+0.04) = 0.04 0.57 = 4 57 ∼= 0.07

(21)

1.11 Solutions for the Illustrated Problems 15 d) P [ N = 1| ¯M ] = P [ ¯M ∩ (N = 1)] P [ ¯M ] = P [(S∪ L) ∩ (N = 1)] P [S∪ L] = P [S∩ (N = 1)] + P [L ∩ (N = 1)] P [S] + P [L] = 0.14 + 0.03 (0.04 + 0.14 + 0.02 + 0.00) + (0.01 + 0.03 + 0.13 + 0.03) = 0.17 0.4 = 17 40 = 0.425

13. a) P [ in = 1| out = 1] = _{P [ out=1}_{|in=1]·P [in=1]+P [ out=1|in=0]·P [in=0]}P [ out=1|in=1]·P [in=1] = _0.7_{·0.5+0.1·0.5}0.7·0.5 = 0.875

b) P [ in = 1| out = X] = _{P [ out=X}_{|in=1]·P [in=1]+P [ out=X|in=0]·P [in=0]}P [ out=X|in=1]·P [in=1] = 0.1·0.5

0.1·0.5+0.1·0.5 = 0.5

P [ in = 0| out = X] = _{P [ out=X}_{|in=1]·P [in=1]+P [ out=X|in=0]·P [in=0]}P [ out=X|in=0]·P [in=0]

= _0.1_{·0.5+0.1·0.5}0.1·0.5 = 0.5 or

P [ in = 0| out = X] = 1 − P [in = 1| out = X] = 1 − 0.5 = 0.5.

c) P [0] + P [1] = 1⇒ 3P [1] + P [1] = 1 ⇒ P [1] = 0.25

P [ in = 1| out = 1] = _{P [ out=1}_{|in=1]·P [in=1]+P [ out=1|in=0]·P [in=0]}P [ out=1|in=1]·P [in=1] ⇒ P [in = 1| out = 1] = 0.7·0.25

0.7·0.25+0.1·0.75 = 0.7

14. First we deﬁne some events as follows:

C = the event that Jack is criminal L = the event that Jack is left handed.

Now we use the Bayes’ rule and write

P [ C| L] = P [L_{P [L]}|C]P [C] = 1×P [C]_{P [L]} = P [C]_{P [L]}

P [L] = P [ L| Cc_{]P [C}c_{] + P [ L}| C]P [C] = (0.2) · (0.4) + (1) · (0.6) = 0.68

P [ C| L] = P [C]_{P [L]} = _0.680.6 ≈ 0.88

15. a) P [red] = P [ red| A]P [A] + P [red| B]P [B] = ₁₀2 · 1₂ + ₁₀6 · 1₂ = 0.4 b) P [ A| red] = P [ red_{P [red]}|A]P [A] = (0.2)_0.4·(0.5) = 0.25

c) Let A & B denote drawing the ﬁrst ball from urn A & B respectively. Then

(22)

P [ A| green] = P [ green| A]P [A]

P [ green| A]P [A] + P [green| B]P [B]

= (0.3)· (0.5)

(0.3)· (0.5) + (0.1) · (0.5) = 0.75

P [ B| green] = P [ green| B]P [B]

P [ green| A]P [A] + P [green| B]P [B]

= (0.1)· (0.5)

(0.3)· (0.5) + (0.1) · (0.5) = 0.25

= ( ₆ 11 ) · (0.75) +( 2 11 ) · (0.25) = 5 11

16. Let us deﬁne the event A↑ to denote drawing a ball from the urn A. Similarly deﬁne another event B↑ for the urn B.

P [win] =(2₈ · 6₇ · 1₂)+(5₆ ·6₈ ·6₇ · 1₂)+(1₈ · 1₇ ·1₂)+(1· 7₈ · 1₇ ·1₂)+0+(7₈ · 1 · 1₇ · 1₂)+ ( 1 6 · 6 7 · 1 2 ) +(6₈ · 5₆ · 6₇ · 1₂)= 0.848 17. a) P [T1T2T3|b] = (0.75)3 = 0.422 b) P [T4|b, T1T2T3] = 0.75 c) P [T4|T1T2T3] = P [T4|b, T1T2T3]P [b|T1T2T3]+P [T4|f, T1T2T3]P [f|T1T2T3] P [T1T2T3] = P [T1T2T3|b]P [b] + P [T1T2T3|f]P [f] = 0.422 × 0.5 + 0.5 × (0.5)3 = 0.2735 P [b|T1T2T3] = P [T_{P [T}1T2₁T_T3₂|b]P [b]_T₃_] = 0.422_0.2735×0.5 = 0.77 P [f|T1T2T3] = 1− P [b|T1T2T3] = 0.23 ⇒ P [T4|T1T2T3] = 0.75× 0.77 + 0.5 × 0.23 = 0.69

d) no, because if T T T happens the probability that the biased coin is chosen increases. e) yes. 18. a) ( 7 3, 2, 1, 1 ) = _(3!)_{·(2!)·(1!)·(1!)}7! = 420

b) For every arrangement that r appears before n, there is a counterpart where n appear before r (just interchange r and n). Thus in half of the arrangements r appears before n. The answer, therefore, is 420₂ = 210. c) The middle letter can be t, r or n. If t, we have 6!_3! = 120 arrangements.

If r (or n), we have _(3!)6!_·(2!) = 60 arrangements. Total = 120 + 60 + 60 = 240.

(23)

1.11 Solutions for the Illustrated Problems 17

Figure 1.4: Tree Diagram for 16

d) We can think of it as “_ X _ X _ X _ X _”, where “X” represents other letters and “_” represents a potential location for “o” (notice that this way consecutive “o”s are avoided). There are 5 locations for “o” and we want to pick three of them. Since order does not matter, the total number of ways is

(

5 2

)

= 10. The other 4 letters (“tmt” have a total of 4!_2! = 12 arrangements among themselves to ﬁll the “X” locations. So the total will be 12× 10 = 120.

(24)

19. a) ( 14 4)( 16 4) (30 8) = 1001_5852925×1820 = 0.31 b) P [all girl] = ( 14 8)( 16 0) (30 8) = _58529253003×1 = 0.000513 P [all boy] = ( 14 0)( 16 8) (30 8) = 1_5852925×12870 = 0.0022 Therefore, P [one gender] = 0.0022 + 0.00051 = 0.00271 c) P [g > b] = ( 14 8 )( 16 0 ) +(14₇)(16₁)+(14₆)(16₂)+(14₅)(16₃) ( 30 8 ) = 3003 + 54912 + 360360 + 1121120 5852925 = 0.263 d) ( 28 6)( 2 2) (30 8) = _5852925376740 = 0.064

20. a) We can look at this question as follows: from the 9 steps, 4 needs to be upward and 5 to be to the right. Therefore, out of 9 steps we want to pick 4 upward ones. We get,

Number of paths =(9₄)= 126.

b) Number of paths from A to C (similar part a) is (4₂) and number of paths from C to B is (5₃). Thus the number of all paths from A to B which pass through C is (4₂)·(5₃). So the required probability

P [C] = ( 4 2)·( 5 3) (9 4) = 6₁₂₆×10 = 0.476. 21. a) if X = +2 if X =−2 HH HT or T H T T Y 0 +1 +2 HH HT or T H T T Y 0 -1 -2 S ={(+2, 0), (+2, +1), (+2, +2), (−2, 0), (−2, −1), (−2, −2)} b) E ={+1, +2}

c) {Y = 0} ={number of heads tossed was 2} d)

(25)

1.11 Solutions for the Illustrated Problems 19 +2 HH HT or TH TT HH HT or TH TT -2 1/2 1/2 1/4 1/2 1/4 1/4 1/2 1/4 (X,Y) Probability (+2,0) 1/8 (+2,+1) 1/4 (+2,+2) 1/8 (-2,0) 1/8 (-2,-1) 1/4 (-2,-2) 1/8 e) P [+2, 0] = 1/8 P [+2, +1] = 1/4 P [+2, +2] = 1/8 P [−2, 0] = 1/8 P [−2, −1] = 1/4 P [−2, −2] = 1/8 f) P [Y = 0] = 1/4 P [Y = +1] = 1/4 P [Y = +2] = 1/8 P [Y =−1] = 1/4 P [Y =−1] = 1/8 g) P [X = 2|Y = 0] = P [X=2,Y =0]_1/4 = 1/2 Similarly, P [X = +2|Y = +1] = 1, P [X = +2|Y = +2] = 1, P [X = +2|Y = −1] = P [X = +2|Y = −2] = 0. 22. a)

P [Path 1 fails] = P [(R1 fails)∪ (R2 fails)]

= P [R1 fails] + P [R2 fails]− P [(R1 fails)∩ (R2 fails)]

= 0.05 + 0.05− (0.05) · (0.05) = 0.0975

P [Path 2 fails] = P [R3 fails] + P [R4 fails]− P [(R3 fails)∩ (R4 fails)]

= 0.08 + 0.08− (0.08) · (0.08) = 0.1536

P [fail] = P [(Path 1 fails)∩ (Path 2 fails)]

(26)

1.12 Drill Problems

Section 1.1,1.2,1.3 and 1.4 - Set theory and

Probability axioms

1. A 6-sided die is tossed once. Let the event A be deﬁned A =‘outcome is a prime number’.

a) Write down the sample space S.

b) The die is unbiased (i.e. all outcomes are equally likely). What is the probability P [A] of event A?

c) Suppose that the die was biased such that: the outcomes 2, 3 and 4 are equally likely; and the outcome 1 is twice as likely as the others. What would have been the probability P [A] of event A?

Ans

a) S ={1, 2, 3, 4, 5, 6} b) P [A] = 0.5

c) P [A] = 3₇

2. An unbiased 4-sided die is tossed. Let the events A and B be deﬁned as:

A =‘outcome is a prime number’ and B ={4}.

a) Find probabilities P [A] and P [B]. b) What is A∩ B? Write down P [A ∩ B].

c) What does this imply about A and B? d) Find P [(A∩ B)c] Ans a) P [A] = 0.5, P [B] = 0.25 b) A∩ B = ϕ, P [A ∩ B] = 0 c) mutually exclusive d) P [(A∩ B)c] = 1

Section 1.6 - Independence

3. An unbiased 4-sided die is tossed. Let the events A and B be deﬁned as:

A =‘outcome is a prime number’ and B =‘outcome is an even number’.

(27)

1.12 Drill Problems 21

b) What is A∩ B? Write down P [A ∩ B]. c) Are A and B mutually exclusive? d) Are A and B independent?

Ans

a) P [A] = 0.5, P [B] = 0.5

b) A∩ B = {2}, P [A ∩ B] = 0.25 c) no

d) yes

4. A pair of unbiased 6-sided dies (X and Y ) is tossed simultaneously. Let the events A and B be denoted as

A: ‘X yields 2’ B: ‘Y yields 2’

a) Write down the sample space S.

Note that each outcome is a pair (x, y), where x, y∈ {1, 2, 3, 4, 5, 6}. b) Write A and B as sets of outcomes. Find corresponding probabilities

P [A] and P [B].

c) What is A∩ B? Write down P [A ∩ B]. d) What is P [A∪ B]?

e) Are A and B mutually exclusive? f) Are A and B independent?

Ans a) S ={(1, 1), (1, 2), (1, 3), (1, 4), (1, 5), (1, 6), (2, 1), (2, 2), (2, 3), (2, 4), (2, 5), (2, 6), (3, 1), (3, 2), (3, 3), (3, 4), (3, 5), (3, 6), (4, 1), (4, 2), (4, 3), (4, 4), (4, 5), (4, 6), (5, 1), (5, 2), (5, 3), (5, 4), (5, 5), (5, 6), (6, 1), (6, 2), (6, 3), (6, 4), (6, 5), (6, 6)} b) A ={(2, 1), (2, 2), (2, 3), (2, 4)}, B ={(1, 2), (2, 2), (3, 2), (4, 2)}, P [A] = 1₆, P [B] = 1₆ c) A∩ B = {(2, 2)}, P [A ∩ B] = ₃₆1 d) P [A∪ B] = 11₃₆ e) no f) yes

(28)

Section 1.5 - Conditional Probability

5. In a certain experiment, A, B, C, and D are events with probabilities P [A] = 1/4, P [B] = 1/8, P [C] = 5/8, and P [D] = 3/8. A and B are disjoint, while

C and D are independent.

Hint: Venn diagrams are helpful for problems like this. a) Find P [A∩ B], P [A ∪ B], P [A ∩ Bc_{], and P [A}_{∪ B}c_].

b) Are A and B independent?

c) Find P [C ∩ D], P [C ∩ Dc], and P [Cc∩ Dc]. d) Are Cc and Dc independent?

e) Find P [A|B] and P [B|A]. f) Find P [C|D] and P [D|C].

g) Verify that P [Cc|D] = 1 − P [C] for this problem. Can you interpret its meaning? Ans a) P [A∩ B] = 0, P [A ∪ B] = 0.375, P [A∩ Bc_{] = 0.25, P [A}_{∪ B}c_{] = 0.875} b) no c) P [C∩ D] = 0.2344, P [C ∩ Dc] = 0.3906, P [Cc∩ Dc] = 0.2344 d) yes e) P [A|B] = 0, P [B|A] = 0 f) P [C|D] = 0.625, P [D|C] = P [D] = 3/8

6. Let A be an arbitrary event. Events D, E and F form an event space.

P [D] = 0.35 P [A|D] = 0.4 P [E] = 0.55 P [A|E] = 0.2 P [F ] = ? P [A|F ] = 0.3

a) Find P [F ] and P [A]. b) Find P [A∩ D].

c) Use Bayes’ rule to compute P [D|A] and P [E|A].

d) Can you compute P [F|A] without using the Bayes’ rule?

e) Compute P [Ac|D], P [Ac|E] and P [Ac|F ]. What is the axiom you had to use?

(29)

g) What theorem(s) you need to compute P [Ac]? Is the value in agreement with P [A] computed in question 10 part a).

Ans

a) P [F ] = 0.1, P [A] = 0.28 b) P [A∩ D] = 0.14

c) P [D|A] = 0.5, P [E|A] = 0.3929 d) P [F|A] = 1 − P [D|A] − P [E|A]

e) P [Ac_{|D] = 0.6, P [A}c_{|E] = 0.8,}

P [Ac|F ] = 0.7

f) P [D|Ac_{] = 0.2917, P [E}_|Ac_{] = 0.6111}

Section 1.7 - Sequential experiments and tree

diagrams

7. Tabulated below is the number of diﬀerent electronic components contained in boxes B1 and B2.

capacitors diodes

B1 3 3

B2 1 5

A box is chosen at random, then a component is selected at random from the box. The boxes are equally likely to be selected. The selection of electronic components from the chosen box is also equally likely.

a) Draw a probability tree for the experiment.

b) What is the probability that the component selected is a diode? c) Find the probability of selecting a capacitor from B1.

d) Suppose that the component selected is a capacitor. What is the prob-ability that it came from B1?

Ans

b) 0.6667 c) 0.25 d) 0.75

(30)

8. Consider the following scenario at the quality assurance division of a certain manufacturing plant.

In each lot of 100 items produced, two items are tested; and the whole lot is rejected if either of the tested items is found to be defective. Outcome of each test is independent of the other tests.

Let q be the probability of an item being defective. Suppose A denotes the event ‘the lot under inspection is accepted’; and k denotes the event ‘the lot has k defective items’, where k ∈ {0, . . . , 100}.

a) Compute probability P [k] of having k defective items in a lot.

b) Find the probability P [A∩ k] that a lot with k defective items is ac-cepted.

Note: check whether your result for P [A∩ k] is intuitive for both k = 0 and k = 99.

c) What is the conditional probability P [A|k] of ‘a lot being accepted’ given it has k defective items?

Ans a) P [k] = (100_k )qk₍₁_−q)100−k b) _{( 98 k ) qk₍₁−q)100−k_{, k} ∈ {0,..,98} 0 , k ∈ {99, 100} c) _{( 1−₁₀₀k )(1−₉₉k), k∈ {0,..,98} 0 , k∈ {99, 100}

9. In a binary digital communication channel the transmitter sends symbols

{0, 1} over a noisy channel to the receiver. Channel introduced errors may

make the symbol received to be diﬀerent from what transmitted.

Let Si ={the symbol i is sent} and Ri ={the symbol i is received}, where

i∈ {0, 1}.

Relevant symbol and error probabilities are tabulated below.

i P [Si] P [R0|Si]

0 0.6 0.9

1 0.4 0.05

a) Draw corresponding probability tree.

(31)

c) Given that a “zero" is received, what is the conditional probability that a “zero" was sent?

d) Given that a “zero" is received, what is the conditional probability that a “one" was sent?

Ans

b) 0.08 c) 0.9643 d) 0.0357

10. In a ternary digital communication channel the transmitter sends symbols

{0, 1, 2} over a noisy channel to the receiver. Channel introduced errors may

i∈ {0, 1, 2}.

i P [Si] P [R0|Si] P [R1|Si]

0 0.6 0.9 0.05

1 0.3 0.049 0.95

2 0.1 0.1 0.1

a) Draw corresponding probability tree.

b) Find the probability that a symbol is received in error.

c) Given that a “zero" is received, what is the conditional probability that a “zero" was sent?

d) Given that a “zero" is received, what is the conditional probability that a “one" was sent?

e) Given that a “zero" is received, what is the conditional probability that a “two" was sent?

Ans

b) 0.095 c) 0.9563 d) 0.026

(32)

11. In a binary digital communication channel the transmitter sends symbols

{0, 1} over a noisy channel to the receiver. Channel introduced errors may

i∈ {0, 1}.

A block of two symbols are sent along the channel. Channel errors on diﬀerent symbol periods can be deemed independent.

i P [Si] P [R0|Si]

0 0.6 0.9

1 0.4 0.05

a) Draw corresponding probability tree (two-stage).

Note: a ‘stage’ corresponds to a single transmitted symbol. b) Find the probability that the block is received in error.

c) Given that ‘00’ received, what is the conditional probability that a ‘00’ was sent?

d) Given that ‘00’ received, what is the conditional probability that a ‘01’ was sent?

Ans

b) 0.1536 c) 0.9298 d) 0.0344

12. A machine produces photo detectors in pairs. Tests show that the first photo detector is acceptable with probability 0.6. When the first photo detector is acceptable, the second photo detector is acceptable with probability 0.85. If the first photo detector is defective, the second photo detector is acceptable with probability 0.35.

Let Ai the event ‘i-th photo detector is acceptable’.

a) Draw a suitable probability tree. b) Describe the event ‘(Ac

1 ∩ A2) ∪ (A1 ∩ Ac2)’ in words. Compute the

corresponding probability.

c) What is the probability P [Ac₁∩ Ac₂] that both photo detectors in a pair are defective?

(33)

1.12 Drill Problems 27 Ans b) P [(Ac 1∩ A2)∪ (A1∩ Ac2)] = 0.74 c) P [Ac 1 ∩ Ac2] = 0.49 d) P [A1|A2] = 0.7846

Section 1.8 - Counting methods

13. A hospital ward contains 15 male and 20 female patients. Five patients are randomly chosen to receive a special treatment. Find the probability of choosing:

a) at least one patient of each gender b) at least two patient of each gender c) all patients from the same gender

d) a group where certain two male patients (say Tim and Joe) are not chosen at the same time

Ans

a) 0.943 b) 0.635 c) 0.057 d) 0.748

14. A bridge club has 12 members (six married couples). Four members are randomly selected to form the club executive. Find the probability that the executive consists of:

a) two men and two women b) all men or all women

c) no married couples d) at least two men

Ans

a) 0.4545 b) 0.0606 c) 0.0303 d) 0.7273

(34)

Figure 1.5: a system that includes both series and parallel subsystems

Section 1.9 - Reliability

15. Figure 1.5 shows a system in a reliability study composed of series and par-allel subsystems. The subsystems are independent. P [W1] = 0.91, P [W2] =

0.87, P [W3] = 0.50, and P [W4] = 0.75. What is the probability that the

system operates successfully?

(35)

Chapter 2 Discrete Random Variables

2.1 Deﬁnitions

Deﬁnition 2.1: A random variable (RV) consists of an experiment with a probability measure P [·] deﬁned on a sample space S and a function that assigns a real number to each outcome in the sample space of the experiment.

Definition 2.2: X is a discrete RV if its range is a countable set: SX = {x1, x2,· · · }. Further, X is a finite RV if its range is a finite set: SX ={x1, x2, . . . , xn}.

2.2 Probability Mass Function

Deﬁnition 2.3: The probability mass function (PMF) of the discrete RV X is deﬁned as

PX(a) = P [X = a].

Theorem 2.1: For a discrete RV with PMF PX(x) and range SX,

1. For any x, PX(x)≥ 0.

2. ∑

x∈SX

PX(x) = 1.

3. For any event B ⊂ SX, P [B] =

∑

x∈B

(36)

2.3 Cumulative Distribution Function (CDF)

Deﬁnition 2.4: The cumulative distribution function (CDF) of a RV X is

FX(r) = P [X ≤ r]

where P [X ≤ r] is the probability that RV X is no larger than r.

Theorem 2.2: For discrete RV X with SX ={x1, x2,· · · }, x1 ≤ x2 ≤ · · ·

• FX(−∞) = 0, FX(∞) = 1.

• If xj ≥ xi, FX(xj)≥ FX(xi).

• For a∈ SX and ϵ > 0, limϵ→0FX(a)− FX(a− ϵ) = PX(a).

• FX(x) = FX(xi) for all x such that xi ≤ x < xi+1.

• For b≥ a, FX(b)− FX(a) = P [a < X ≤ b]

2.4 Families of Discrete RVs

Deﬁnition 2.5: X is Bernoulli(p) RV if the PMF of X has the form

PX(x) =        1− p, x = 0 p, x = 1 0, otherwise , with SX ={0, 1}.

Deﬁnition 2.6: X is a Geometric(p) RV if the PMF of X has the form

PX(x) =    p(1− p)x−1_, _{x = 1, 2, . . .} 0, otherwise ,

(37)

2.5 Averages 31

Deﬁnition 2.7: X is Binomial(n, p) RV if the PMF of X has the form

PX(x) =    ( n x ) px(1− p)n−x, x = 0, 1, 2, . . . , n 0, otherwise ,

where 0 < p < 1 and n is an integer with n ≥ 1.

Deﬁnition 2.8: X is Pascal(k, p) RV (also known as negative binomial RV)

if the PMF of X has the form

PX(x) =    ( x−1 k−1 ) pk₍₁_{− p)}x−k_, _{x = k, k + 1, k + 2, . . .} 0, otherwise ,

where 0 < p < 1 and k is an integer such that k ≥ 1.

Deﬁnition 2.9: X is Discrete Uniform(k, l) RV if the PMF of X has the

form PX(x) =    1 l−k+1, x = k, k + 1, k + 2, . . . , l 0, otherwise ,

where the parameters k and l are integers such that k < l.

Deﬁnition 2.10: X is Poisson(α) RV if the PMF of X has the form

PX(x) =    αx_e−α x! , x = 0, 1, 2, . . . 0, otherwise , where α > 0.

2.5 Averages

Deﬁnition 2.11: A mode of X is a number xmod satisfying PX(xmod)≥ PX(x)

for all x.

Deﬁnition 2.12: A median of X is a number xmedsatisfying P [X < xmed)] =

(38)

Deﬁnition 2.13: The mean (aka expected value or expectation) of X is E[X] = µX = ∑ x∈SX xPX(x). Theorem 2.3:

1. If X ∼ Bernoulli(p), then E[X] = p. 2. If X ∼ Geometric(p), then E[X] = 1/p. 3. If X ∼ Poisson(α), then E[X] = α. 4. If X ∼ Binomial(n, p), then E[X] = np. 5. If X ∼ Pascal(k, p), then E[X] = k/p.

6. If X ∼ Discrete Uniform(k, l), then E[X] = (k + l)/2.

2.6 Function of a Random Variable

Theorem 2.4: For a discrete RV X, the PMF of Y = g(X) is

PY(y) = P [Y = y] =

∑

x:g(x)=y

PX(x)

i.e., P [Y = y] is the sum of the probabilities of all the events X = x for which

g(x) = y.

2.7 Expected Value of a Function of a Random

Variable

Theorem 2.5: Given X with PMF PX(x) and Y = g(X), the expected value

of Y is

E[Y ] = µY = E[g(X)] =

∑

x∈SX

(39)

2.8 Variance and Standard Deviation 33

2.8 Variance and Standard Deviation

Deﬁnition 2.14: The variance of RV X is VAR[X] = σ_X2 = E[(X − µX)2],

VAR[X] = ∑

x∈SX

(x− µX)2PX(x)≥ 0.

Equivalently, the expected value of Y = (X − µX)2 is VAR [X].

Deﬁnition 2.15: The standard deviation of RV X is σX =

√

VAR[X].

Theorem 2.6:

VAR[X] = E[X2]− (E[X])2

Theorem 2.7: For any two constants a and b, VAR[aX + b] = a2VAR[X]

Theorem 2.8:

1. If X ∼ Bernoulli(p), then VAR[X] = p(1 − p). 2. If X ∼ Geometric(p), then VAR[X] = (1 − p)/p2_.

3. If X ∼ Binomial(n, p), then VAR[X] = np(1 − p). 4. If X ∼ Pascal(k, p), then VAR[X] = k(1− p)

p2 .

5. If X ∼ Poisson(α), then VAR[X] = α.

6. If X ∼ Discrete Uniform(k, l), then VAR[X] = (l− k)(l − k + 2)

12 .

Deﬁnition 2.16: For RV X,

(a) The n-th moment is E[Xn_]

(b) The n-th central moment is E[(X− µX)n].

2.9 Conditional Probability Mass Function

Deﬁnition 2.17: Given the event B, with P [B] > 0, the conditional proba-bility mass function of X is

(40)

Theorem 2.9: For B ⊂ SX, PX|B(x) = P [X = x, B] P [B] =    P [X=x] P [B] , x∈ B 0, otherwise.

Deﬁnition 2.18: The conditional expected value of RV given condition is

E[X|B] = µX|B =

∑

x∈B

xPX|B(x).

Theorem 2.10: The conditional expected value of Y = g(X) given condition

B is

E[Y|B] = µY|B =

∑

x∈B

g(x)PX|B(x).

2.10 Basics of Information Theory

Deﬁnition 2.19: The information content of any event A is deﬁned as

I(A) =− log₂P [A]

This deﬁnition is extended to a Random Variable X.

Deﬁnition 2.20: The information content of X is deﬁned as

I(X) = −E[log₂(P [X = x])]

=−∑

x

PX(x) log2PX(x)

I(X) is measured in bits. Suppose X produces symbols s1, s2, ...sn. A binary code

is used to represent the symbols Let li bits used represent si, for i = 1...n .

Deﬁnition 2.21: The average length of the code is

E[L] =∑

i

pili

Definition 2.22: The efficiency of the code is defined as

η = I(X)

(41)

Theorem 2.11: Huﬀman’s Algorithm

1. Write symbols in decreasing order with their probabilities. 2. Merge in pairs from the bottom and reorder.

3. Repeat until one symbol is left. 4. Code each branch with "1" or "0".

2.11 Illustrated Problems

1. Two transmitters send messages through bursts of radio signals to an an-tenna. During each time slot each transmitter sends a message with prob-ability 1/2. Simultaneous transmissions result in loss of the messages. Let

X be the number of time slots until the ﬁrst message gets through. Let Ai

be the event that a message is transmitted successfully during the i-th time slot.

a) Describe the underlying sample space S of this random experiment (in terms of Ai andAci) and specify the probabilities of its outcomes.

b) Show the mapping from S to SX, the range of X.

c) Find the probability mass function of X.

d) Find the cumulative distribution function of X.

2. An experiment consists of tossing a fair coin until either three heads or two tails have appeared (not necessarily in a row). Let X be the number of tosses required.

a) Describe the underlying sample space S of this random experiment using a tree diagram and specify the probabilities of its outcomes. b) Show the mapping from S to SX, the range of X.

c) Find the probability mass function of X.

d) Find the cumulative distribution function of X.

3. Ten balls numbered from 1 to 10 are in an urn. Four balls are to be chosen at random (equally likely) and without replacement. We deﬁne a random variable X which is the maximum of the four drawn balls (e.g., if the drawn balls are numbered 3, 2, 8 and 6, then X = 8).

a) What is the range of X, SX?

b) Find the PMF of X and plot it.

c) Find the probability that X be greater than or equal to 7.

4. The Oilers and Sharks play a best out 7 playoﬀ series. The series ends as soon as one of the teams has won 4 games. Assume that Sharks (Oilers)

(42)

are likely to win any game with a probability of 0.45(0.55) independently of any other game played. For n = 4, 5, 6, 7 deﬁne events On= {Oilers win the

series in n games} and Sn = {Sharks win the series in n games}.

a) Suppose the total number of games played in the series is N . Describe the event {N = n} in terms of On and Sn and ﬁnd the PMF of N .

b) Let W be the number of Oilers wins in the series. Now, express the events {W = n} for n = 0, 1, . . . , 4 in terms of On and Sn and ﬁnd the

PMF of W .

5. The random variable X has PMF

PX(x) =    k_xcx+12₊₁, x =−2, −1, 0, 1, 2 0, otherwise

a) Find the value of k and the range of c for which this is a valid PMF. b) For c = 0 and k found in part a, compute and plot the CDF of X.

c) Compute the mean and the variance of X. 6. The CDF of a random variable is as follows

FX(a) =                    r, a < 1 0.3, 1≤ a < 3 s, 3≤ a < 4 0.9, 4≤ a < 6 t, 6≤ a

a) What are the values of r and t and the valid range of s? b) What is P [2 < X ≤ 5]?

c) Knowing that P [X = 3] = P [X = 4], Find s and plot the PMF of X. 7. Studies show that 20% of people are left handed. Also, it is known that 15%

of people are allergic to dust.

a) What is the probability that in a class of 40 students, exactly 8 students be left handed?

b) Assuming that being left handed is independent of being allergic to dust, what is the probability that in a class of 30 students more than 2 students be both left handed and allergic to dust?

c) To study a new allergy medicine, the goal is to select a group of 10 people that are allergic to dust. Randomly selected people are tested to check whether or not they are allergic to dust. What is the probability that after testing exactly 75 people, the needed group of 10 is found?

(43)

8. A game is played with probability of win P [W ] = 0.4. If the player wins 10 times (not necessarily consecutive) before failing 3 times (not necessarily consecutive), a $100 award is given. What is the probability that the award is won? Hint: Identify all award-winning cases (10W, 10W + 1F, 10W + 2F ) and notice that all award-winning cases ﬁnish with a W .

9. Phone calls received on a cell phone are totally random in time. Therefore (as we proved in class), the number of telephone calls received in a 1 hour period is a Poisson random variable. If the average number of calls received during 1 hour is 2 (meaning that α = 2) answer the following questions:

a) What is the probability that exactly 2 calls are received during this one hour period?

b) The cell phone is turned oﬀ for 15 minutes, what is the probability that no call is missed.

c) What is the probability that exactly 2 calls are received during this one hour period and both calls are received in the ﬁrst 30 minutes?

d) Find the standard deviation of the number of calls received in 15 min-utes.

10. A stop-and-wait protocol is a simple network data transmission protocols in which both the sender and receiver participate. In its simplest form, this protocol is based on one sender and one receiver. The sender establishes the connection and sends data in packets. Each data packet is acknowl-edged by the receiver with an acknowledgement packet. If a negative ac-knowledgement arrives (i.e., the received packet contains errors), the sender retransmits the packet.

Now consider the use of this protocol on a network with packet error rate 1/70 (acknowledgement packets are assumed to receive perfectly). Let X be the number of transmissions necessary to send one packet successfully.

a) Find the probability mass function of X. b) Find the mean and variance of X.

c) If successful transmission does not take place in 12 attempts, the sender declares a transmission failure. Find the probability of a transmission failure.

d) Assume that 100 packets are to be transmitted. Let Y be the number of transmissions necessary to send all 100 packets. Find the probability mass function of Y .

e) Find the mean and variance of Y .

(44)

12. The random variable X has PMF PX(x) =    c/(1 + x2), x =−3, −2, . . . , 3 0, otherwise a) Compute FX(x).

b) Compute E[X] and VAR [X].

c) Consider the function Y = 2X2_{. Find P}

Y(y).

d) Compute E[Y ] and Var[Y ].

13. Consider a source sending messages through a noisy binary symmetric chan-nel (BSC); for example, a CD player reading from a scratched music CD, or a wireless cellphone capturing a weak signal from a relay tower that is too far away.

For simplicity, assume that the message being sent is a sequence of 0’s and 1’s. The BSC parameter is p. That is, when a 0 is sent, the probability that a 0 is (correctly) received is p and the probability that a 1 is (incorrectly) received is 1− p. Likewise, when a 1 is sent, the probability that a 1 is (correctly) received is p and the probability that a 0 is (incorrectly) received is 1− p.

Let p = 0.97 for the BSC. Suppose the all-zero byte (i.e. 8 zeros) is trans-mitted over this channel. Let X be the number of 1s in the received byte.

a) Find the probability mass function of X. b) Compute E[X] and Var[X].

c) Suppose that in all transmitted bytes, the eighth bit is reserved for parity (even parity is set for the whole byte), so that the receiver can perform error detection. Let E be the event of an undetectable error. Describe E in terms of X. Find P [E].

14. PX(x) =    c/(1 + x2_), _{x =}_{−3, −2, . . . , 3} 0, otherwise

a) Deﬁne event B ={X ≥ 0}. Compute PX|B(x).

b) Compute FX|B(x).

c) Compute E[X|B] and Var[X|B].

15. Let X be a Binomial(8, 0.3) random variable. a) Find the standard deviation of X. b) Deﬁne B={X is odd}. Find PX|B(x).

(45)

c) Find E[X|B].

d) Find Var[X|B].

2.12 Solutions for the Illustrated Problems

1. a) Ai: one of them sends message at ith time slot, P [Ai] = 1₄ +1₄ = 1₂

Ac

i: both or none of them sends message at ith time slot, P [Aci] = 12 S ={A1, Ac1A2, Ac1Ac2A3, . . . , Ac1Ac2· · · Acn−1An, . . .} b) S 1 A 2 1A Ac 3 2 1A A Ac c n c n c c A A A A₁ ₂ ₋₁ X S 1 2 3 n c) PX(t) =    (1/2)t_, _t∈ {1, 2, . . .} 0, otherwise d) FX(t) =          0, t < 1 .. . 1 2 + 1 4 +· · · + 1 2n−1 = 1− 1 2n−1, n− 1 ≤ tt < n or FX(t) =    0, t < 1 FX(t− 1) + ( 1 2 )n−1 , n− 1 ≤ t < n 2. a)

(46)

H T H T H T H T H T H T H T H T H T 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 P [HHH] = 1/8, P [HHT H] = P [HT HH] = P [HT HT ] = 1/16, P [HT T ] = 1/8 P [T HHH] = P [T HHT ] = 1/16, P [T HT ] = 1/8, P [T T ] = 1/4, P [HHT T ] = 1/16 b) S TT HHH HTT THT HHTH, HTHH, HTHT, HHTT, THHH, THHT X S 2 3 4 c) PX(t) =              1/4, t = 2 3/8, t = 3 3/8, t = 4 0, otherwise d) FX(t) =              0, t < 2 1/4, 2≤ t < 3 5/8, 3≤ t < 4 1, t≥ 4 3. a) SX ={4, 5, 6, 7, 8, 9, 10}

(47)

b) The probability that x = n means one of these four balls is n and the other three are chosen form n− 1 balls with number less than n.

P [X = n] = ( n−1 3 )( 1 1 ) ( 10 4 ) = (n− 1).(n − 2).(n − 3) 1260 P [X = 4] = ₂₁₀1 = 0.0048 P [X = 5] = 4×3×2₁₂₆₀ = 0.019 P [X = 6] = 5×4×3₁₂₆₀ = 0.048 P [X = 7] = 6×5×4₁₂₆₀ = 0.095 P [X = 8] = 7×6×5₁₂₆₀ = 0.167 P [X = 9] = 8×7×6₁₂₆₀ = 0.267 P [X = 10] = 9×8×7₁₂₆₀ = 0.4 c) P [X ≥ 7] = 0.095 + 0.167 + 0.267 + 0.4 = 0.929

4. a) {N = n} is the event that the series ends in n games. This means either Sn or On occurs: {in n − 1 games, Sharks win 3 times (and

Oilers win n− 4 times) and in the nth _{game, Sharks win} or {in n}− 1

games, Oilers win 3 times (and Sharks win n− 4 times) and in the nth game, Oilers win}.

{N = n} = On∪ Sn P [N = n] =(n−1₃ )· (0.45)4_(0.55)n−4₊(n−1 3 ) · (0.45)n−4_(0.55)4 b) {W = 0} = S4, {W = 1} = S5, {W = 2} = S6, {W = 3} = S7 {W = 4} = O4+ O5+ O6+ O7 P [W = 0] = (0.45)4 _{= 0.041} P [W = 1] =(4₃)(0.45)4_{· (0.55) = 0.09} P [W = 2] =(5₃)(0.45)4· (0.55)2 = 0.124 P [W = 3] =(6₃)(0.45)4_{· (0.55)}3 _{= 0.136} P [W = 4] = 0.608 5. a) ∑PX(x) = 1⇒ k [ 1−2c 5 + 1−c 2 + 1 + 1+c 2 + 1+2c 5 ] = 1⇒ k = 5 12 PX(−2) ≥ 0 ⇒ 1 − 2c ≥ 0 ⇒ c ≤ 0.5 PX(2)≥ 0 ⇒ 1 + 2c ≥ 0 ⇒ c ≥ −0.5 thus,−0.5 ≤ c ≤ 0.5 b) FX(x) =                          0, x <−2 1/12, x =−2 7/24, x =−1 17/24, x = 0 22/24, x = 1 1, x≥ 2

ECE342 Course Notes

ECE342- Probability for

Electrical & Computer Engineers

C. Tellambura and M. Ardakani

Contents

Chapter 1

Basics of Probability Theory

Goals of EE387

1.1

Set theory

1.1.1

Basic Set Operations

1.1.2

Algebra of Sets

1.2

Applying Set Theory to Probability

1.3

Probability Axioms

1.4

Some Consequences of Probability Axioms

1.5

Conditional probability

1.6

Independence

1.7

Sequential experiments and tree diagrams

1.8

Counting Methods

1.9

Reliability Problems

1.10

Illustrated Problems

0

1

1

X

0

Input

Output

1.11

Solutions for the Illustrated Problems

1.12

Drill Problems

Section 1.1,1.2,1.3 and 1.4 - Set theory and

Probability axioms

Section 1.6 - Independence

Section 1.5 - Conditional Probability

Section 1.7 - Sequential experiments and tree

diagrams

Section 1.8 - Counting methods

Section 1.9 - Reliability

Chapter 2

Discrete Random Variables

2.1

Deﬁnitions

2.2

Probability Mass Function

2.3

Cumulative Distribution Function (CDF)

2.4

Families of Discrete RVs

2.5

Averages

2.6

Function of a Random Variable

2.7

Expected Value of a Function of a Random

Variable

2.8

Variance and Standard Deviation

2.9

Conditional Probability Mass Function

2.10

Basics of Information Theory

2.11

Illustrated Problems

2.12

Solutions for the Illustrated Problems