PROBABILITY AND STOCHASTIC PROCESSES FOR ENGINEERS. Dr. Dominic Ho Department of Electrical Engineering & Computer Science University of Missouri

ENGINEERS

Dr. Dominic Ho

Department of Electrical Engineering & Computer Science University of Missouri

Ch.2 Probability Theory

2.1 Introduction

A random event is a “large” or visible effect with a “small” invisible, or nonexistent cause.

The probability of an event is an indication of how likely it is to occur. Eg. A random experiment is performed N times, n of which result in the

occurrence of event A. Then

P (A) = lim

N −>∞

n

N. (1)

2.2 Fundamental Definition and Sets 2.2.1 Fundamental definition

• Sample Space Ω: A set each of its element represents one possible result. • Sample Points or Outcomes: elements of sample space.

Eg. An experiment of flipping two coins has Ω = { (H, H) , (H, T ) , (T, H) , (T, T ) } (2) F = { {(H, H)}, {(H, T )}, · · · , { (H, H), (H, T )}, · · · {(H, H), (H, T ), (T, H), (T, T )} } (3) size of F = C₁4 + C₂4 + C₃4 + C₄4 = 24. Outcomes = (H, H) , (H, T ) , (T, H) , (T, T ).

E = {(H, H), (H, T )} is the event that a head appears in the first coin. Obviously, E ∈ F .

2.2.1 Set notation and operations

Notation Venn Diagram (i) Element: ω ∈ A

(ii) Union: A ∪ B = {ω ∈ Ω : ω ∈ A or ω ∈ B}

(iii) Intersection: A ∩ B = {ω ∈ Ω : ω ∈ A and ω ∈ B}

(iv) Complement: Ac = {ω ∈ Ω : ω not ∈ A}

(v) Inclusion (Subset): A ⊂ B, if ω ∈ A, then ω ∈ B

Operations

(i) DeMorgan’s Law

(A ∪ B ∪ C)c = Ac ∩ Bc ∩ Cc (4)

(ii) Distributive Laws

A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ C) (5)

2.3 The Axioms of Probability

Defn.: let Ω be a sample space. A probability measure or probability P is an assignment of a real number P (A) to each event A ⊆ Ω (or A ∈ F ) that satisfies

1. P (φ) = 0, P (Ω) = 1. (φ=empty set) 2. 0 ≤ P (A) ≤ 1.

3. For disadjoint events A and B (i.e. A ∩ B = φ), P (A ∪ B) = P (A) + P (B).

Properties

1. For equally likely outcomes,

P (A) = no. of elements in A

no. of elements in Ω. (7)

Proof: P (B) = P ((B \ A) ∪ A) = P (B \ A) + P (A), since (B \ A) ∩ A = φ. Hence P (B \ A) = P (B) − P (A). 3. P (Ac) = 1 − P (A). 4. if A ∩ B 6= φ, P (A ∪ B) = P (A) + P (B) − P (A ∩ B). (8) Proof:

5. If A1, A2, · · · , An are disjoint events,

P (∪n_i=1Ai) = n X

6. In general, if A1, A2, · · · , An are not disjoint, P ( ∪n_i=1Ai) = n X i=1P (Ai) − X i₁<i₂P (Ai1 ∩ Ai2) + X i₁<i₂<i₃ P (Ai1 ∩ Ai2 ∩ Ai3) − · · · + (−1) n+1 P (∩n_i=1Ai). (10)

Eg. Suppose Ω = {ω1, ω2, ω3, ω4}, P (ω1, ω2, ω3) = 1/3 , P (ω1, ω2) = 1/6 and P (ω1, ω3, ω4) = 5/6. Find Pi = P (ωi), i = 1, 2, 3, 4. Sol. P1 + P2 + P3 + P4 = P (Ω) = 1 (11) P1 + P2 + P3 = 1 3 (12) P1 + P2 = 1 6 (13) P₁ + P₃ + P₄ = 5 6 (14)

4 equations with 4 unknowns. Solving gives P1 = 0, P2 = 1/6, P3 = 1/6,

Eg. Suppose we toss two fair coins. Let E be the event that the first coin falls heads, and F be the event that the second coin falls heads. Fine

P (E ∪ F ). Sol.

2.4 Conditional Probability

Eg. An urn contains 10 white, 5 yellow and 10 black marbles. A marble is chosen at random, find (i) the probability that it is yellow, (ii) the

probability that it is yellow, when it is noted that it is not black. Sol. (i) P (yellow) = 5 10 + 5 + 10 = 1 5. (15) (ii)

P (yellow | not black) = no. of yellow

no. of yellow + no. of white =

5

5 + 10 = 1

3. (16)

Defn.: For any event A and B with P (B) > 0, the conditional probability of A given B is

P (A | B) = P (A ∩ B)

Eg. Roll two dice and observe the numbers showing upper most on each. Given that the two do not show identical results, what is the probability that the sum is 7?

Sol.

Let A=”sum is 7” and B=”non-identical result”. We need to find P (A | B) = P (A ∩ B)/P (B).

Basic rules of conditional probability 1. P (A ∩ B) = P (A | B) P (B). 2. P (B | A) = P (A ∩ B) P (A) = P (A | B) P (B) P (A) . (18)

3. If A1, A2, · · · ∈ F and Ai ∩ Aj = φ ∀i 6= j. Then

P (A₁ ∪ A₂ ∪ · · · | B) = X i P (Ai | B). (19) Proof: P (A1 ∪ A2 · · · | B) = P ((A1 ∪ A2 ∪ · · ·) ∩ B) P (B) = P ((A1 ∩ B) ∪ (A2 ∩ B) ∪ · · ·) P (B) , distributive law = P i P (Ai ∩ B) , since (A ∩ B) ∩ (A ∩ B) = φ

= X i P (Ai | B). (20) 4. If A1, A2, · · · ∈ F , Ai ∩ Aj = φ ∀i 6= j, and B ⊂ (A1 ∪ A2 ∪ · · ·), then X i P (Ai | B) = 1. (21) Proof:

Since B ⊂ (A1 ∪ A2 ∪ · · ·), let A1 ∪ A2 ∪ · · · = B ∪ X, where

X = (A1 ∪ A2 ∪ · · ·) \ B. Obviously, B ∩ X = φ. Thus X i P (Ai | B) = P (A1 ∪ A2 ∪ · · · | B) = P ((B ∪ X) | B) = P ((B ∪ X) ∩ B) P (B) = P ((B ∩ B) ∪ (X ∩ B)) P (B) = P (B ∩ B) + P (X ∩ B) P (B) = P (B) + 0 P (B) = 1. (22)

5. Product rule: if A1, A2, · · · , An ∈ F , then

P (An ∩ An−1 · · · ∩ A1) = P (An | An−1 ∩ An−2 · · · ∩ A1)

• P (A_n−1 | A_n−2 ∩ A_n−3 · · · ∩ A₁) ...

2.5 Law of Total Probability and Bayes’ Theorem

Theorem: Let A1, A2, · · · , An ∈ F be pairwise disjoint and P (Ai) 6= 0 ∀i.

If B ∈ F satisfies P (B) 6= 0 and B ⊂ ∪n_j=1Aj, then we have

1. Law of Total Probability:

P (B) = Xn i=1P (Ai)P (B|Ai). (24) 2. Bayes’ Theorem: P (Ai | B) = P (B|Ai)P (Ai) Pn j=1 P (Aj)P (B|Aj) , for any i = 1, 2, · · · , n. (25) Proof:

1. Law of Total Probability:

Since Ai are pairwise disjoint and B ⊂ ∪n_i=1Ai, it is clear

B = (B ∩ A1) ∪ (B ∩ A2) ∪ · · · ∪ (B ∩ An) = ∪n_i=1(B ∩ Ai) (26)

and all B ∩ Ai are pairwise disjoint. Thus

P (B) = Xn

i=1P (B ∩ Ai) = n X

2. Bayes’s Rule: P (Ai|B) = P (Ai)P (B|Ai) P (B) , basic rule 2 = P (Ai)P (B|Ai) Pn j=1 P (Aj)P (B|Aj)

, law of total prob. (28)

Eg. Three boxes contains R, B and Y balls. Box A1 has 2 R, 5 B and 1 Y.

Box A2 has 3 R, 2 B and 2 Y. Box A3 has 1 R, 2 B and 6 Y. A ball is

chosen from a randomly selected box and the ball is R. Find the probability that the ball is from Box A₂.

Eg. Cancer test:

A = the test states a patient has cancer, B = the patient has cancer,

Ac = the test states the patient has no cancer, Bc = the patient does not have cancer.

Given P (A|B) = P (Ac|Bc) = 0.95 and P (B) = 0.005. Is the test good? Sol.

Eg. I am invited to have dinner with nine people, 2 are nice and good

looking girls and the rest are guys. We shall be assigned seats randomly in a round table. What is the probability that I will sit beside a girl? I will go if the probability is greater than 0.3. Shall I go?

2.6 Statistical Independence

Defn.: Two events A and B are independent if and only if

P (A ∩ B) = P (A)P (B). (29)

Equivalently,

P (A|B) = P (A) or P (B|A) = P (B). (30)

In general, A1, A2, · · · , An ∈ F are independent if and only if

P (∩i∈IAi) = Y

i∈I P (Ai) ,

∀I ⊂ {1, 2, · · · , n}. (31) Eg. A, B and C are independent if and only if

P (A ∩ B) = P (A)P (B) , P (A ∩ C) = P (A)P (C) , P (B ∩ C) = P (B)P (C) ,

P (A ∩ B ∩ C) = P (A)P (B)P (C). (32)

Theorem: If A1, A2, · · · , An are independent, each possible combination of

Eg. A window is made with three layers of glass. The outside layer cracks with probability 0.2, the inner layer with probability 0.3 and the middle layer with probability of 0.15. Assumming the three layers behave independently, what is the probability of no layer cracks? That at least one layer cracks? Sol.

2.7 Bernoulli Trials and Compound Experiment

Defn.: Independent trials that result in a success with probability p and a failure with probability 1 − p are called Bernoulli trials.

Eg. Flipping a coin with Head denoting success and Tail denoting fail is a Bernoulli trial. P (success) = 1/2, P (fail) = 1/2.

Defn.: A compound experiment is a random experiment which consists of a sequence of M independent trials of a simple experiment.

Let A be an event of success in a single Bernoulli trial, P (A) = p and

P (Ac) = 1 − p. Then the sample space of the compound experiment after M trials is

Ω_M = {(A, A, · · · , A)

| {z }

M

, (Ac, A, · · · , A), · · · , (A, Ac, A, · · · , A), · · ·

(Ac, Ac, A, · · · , A), · · · , (Ac, Ac, · · · , Ac)}. (33) Size of ΩM = C₀M + C₁M + · · · + C_MM = 2M.

Define (M+1) events as

P (Ai) = {i occurrences of A} ∈ Ω_M, i = 0, 1, · · · , M. (34) Then

P (Ai) = (no. of arngmts of i successes in M trials) × P (success)iP (fail)M −i = C_iM pi(1 − p)M −i , i = 0, 1, 2, · · · , M. (35)

P (Ai), i = 0, 1, 2, · · · , M is called the Binomial distribution.

Eg. The probability of obtaining 5 Heads when throwing a coin 10 times is

P (5 Heads) = C₅10    1 2    5    1 2    5 = 252 210 = 0.2461. (36)

Eg. Fire 5 missiles to destroy a target. It is gone if at least two missiles hit the target. P (hit) = 0.1, P (miss) = 0.9. Find P (target is not destroyed). Sol.

Generalization:

Consider M -independent trials. Each trial has k possible outcomes

A1, A2, · · · , Ak, and P (Ai) = pi. Then the probability that the M trials

has r1 of A1, r2 of A2, · · ·, rk of Ak is

M !

r1!r2! · · · rk!

(p1)r1(p2)r2 · · · (pk)rk. (37)

Eg. Emergency calls,

P (Ambulance) = 0.25, P (Police) = 0.6 and P (Fire Fighter) = 0.15. Find P (3A’s, 5P’s and 2FF’s in 10 calls).

2.8 A Communication Example Problem: m {m ,m , ...,m } r {r ,r , ...,r } m(r) 0 1 M-1 0 1 J-1 channel receiver

message received values received symbol

A priori probability: P (m₀), P (m₁), · · · , P (m_{M −1}).

Channel specified by P (rj|mi) = P (receiving rj given mi was transmitted).

The receiver assigns each rj a transmitted symbol, i.e.

ˆ

m{ro, r1, · · · , rJ −1} → {m0, m1, · · · , mM −1}. (38)

Solution:

P (correct transmission) = P (transmitted symbol= received symbol) = P (m = ˆm(r))

= XJ

j=1P (m = ˆm(r) | r = rj) P (r = rj). (39)

Since P (r = rj) = +ve and is independent of the assignment rule,

maximize P (correct transmission) ⇔ maximize P (m = ˆm(r)|r = rj),

∀j = 0, 1, 2, ..., J − 1. (40) But P (m = ˆm(r)|r = rj) = P (r = rj|m = ˆm(rj)) P (m = ˆm(rj)) P (r = rj) . (41) Thus

max. P (correct transmission) ⇔ max. P (r = rj|m = ˆm(rj)) P (m = ˆm(rj)),

Eg. m = {0, 1} and r = {a, E, b}.

Find the optimum assignment rule ˆm{a, E, b} → {0, 1} for the following channel: 0 1 a E b 1/2 1/4 1/8 5/8 1/3 2/3 1/4 1/4 P(a|0)=1/2, etc.