ACF for AR(p) and Linear Difference Equations

When computing the ACF for MA processes, we start from covariances and we work our way from there. By starting from first principles, we call this the direct method. We can also use direct method to compute the ACF for AR processes. However because of the recursive nature of these processes, direct method is tedious. Instead, we use more efficient methods to compute the ACF for AR processes such as through linear processes and solving linear difference equations.

Linear process method

Through any of the 3 inversion methods, we can write the AR(1) process with |φ| < 1 as a linear process

By Theorem 3.2, we can write the autocovariance function as γ(h) = σ²

The infinite sum is a geometric series. Since |φ| < 1, we can use the formula P∞ i=0rⁱ = 1/(1 − r) to write

∞

X

j=0

φ^2j = 1 1 − φ². Hence the ACVF for AR(1) is

γ(h) = σ²φ^|h|

1 − φ², and its variance is

γ(0) = σ² 1 − φ². Consequently, the ACF of AR(1) is

ρ(h) = γ(h)

γ(0) = φ^|h|.

To use the linear process method, we need to first use the 3 inversion methods to get the linear process representation of any weakly stationary models, then we apply Theorem 3.2 to compute the ACF. For higher order AR or ARMA processes, finding their linear process representation can be very complicated. For these models, we take advantage of their recursive nature by computing ACFs through solving linear difference equations.

Linear difference equations

To illustrate this method, let us compute the ACF for an AR(2) process:

X_t = φ₁X_t−1+ φ₂X_t−2+ Z_t. (3.10) Let us assume that this process is causal and hence weakly stationary. Under AR models, time series is related recursively such that X_t depends on its previous value X_t−1 and X_t−2. The idea is that let us also try to relate the ACFs in such a way. Because if ACFs are related with each other recursively, we can solve this recursion to compute ρ(h).

In the AR(2) model, let us take covariances on both sides with respect to X_t−h with h > 0,

Cov(X_t, X_t−h) = φ₁Cov(X_t−1, X_t−h) + φ₂Cov(X_t−2, X_t−h) + Cov(Z_t, X_t−h).

Since X_t−h is weakly stationary, we can write it as a linear process and the last term is Cov(Z , X ) = Cov Z,

∞

Xψ Z

!

= 0.

Using the definition of weakly stationarity, we can write the rest of the terms as γ(h) = φ₁γ(h − 1) + φ₂γ(h − 2), h = 2, 3, . . . .

Divide both sides by γ(0) to get

ρ(h) − φ₁ρ(h − 1) − φ₂ρ(h − 2) = 0, h = 2, 3, . . . . (3.11) Hence we related the ACFs recursively. This is an example of a linear difference equation.

When something is a recursion, we need to start somewhere. For example, we need to have values for ρ(1) and ρ(0) to jump start the recursion above. These values are called the initial values for the linear difference equation. We know that ρ(0) = 1. To get ρ(1), set h = 1 in the recursion above. Since ρ(1) = ρ(−1), we obtain ρ(1) = φ₁/(1 − φ₂).

The linear difference equation above is not very practical to use when we need to compute ρ(h) for a large h, e.g., ρ(100). To do this, we need to recursively compute ρ(h) starting from the initial values. Instead, we would like to have a formula for ρ(h) that will give us the direct answer by substituting different values of h. We get this formula by solving this linear difference equation. In the next section, let us take a detour to learn how to solve this type of equations.

Solving linear difference equations

Definition 3.12. A function a(n), n = d, d + 1, . . . satisfies an order d homogeneous linear difference equation if

a(n) = c₁a(n − 1) + c₂a(n − 2) + · · · + c_da(n − d), (3.12) where the c_i’s are constants, and c_d6= 0. To kick start the recursion, we have initial values for a(d − 1), a(d − 2), . . . , a(0). These initial values with the linear difference equation above completely determine the entire series.

Using the backshift operator, we can write

a(n) = c₁Ba(n) + c₂B²a(n) + · · · + c_dB^da(n) (1 − c₁B − c₂B²− · · · − c_dB^d)

| {z }

call this φ(B)

a(n) = 0

φ(B)a_n = 0.

Substituting B with z ∈ C, we can write the characteristic polynomial for the linear difference equation as

φ(z) = 1 − c₁z − c₂z²− · · · − c_dz^d. (3.13) The characteristic polynomial contains important info about the linear difference equation.

One of them is how to solve these equations.

Find the roots of this polynomial r₁, r₂, . . . , r_d. Then if the roots are all distinct, the solution for the linear difference equation is

a(n) = k₁r₁⁻ⁿ+ k₂r⁻ⁿ₂ + k₃r₃⁻ⁿ+ k₄r⁻ⁿ₄ + · · · + k_dr_d⁻ⁿ. (3.14) The values for k₁, k₂, . . . , k_d are obtained by substituting the initial values to the solution above and solving the resulting simultaneous equations.

If there are multiple roots, e.g., r₂ = r₁ = r, we instead write By (3.13), the characteristic polynomial is

φ(z) = 1 − z − z².

Because they are distinct, we use (3.14) to write

F_n = k₁ −1 −√

Example 3.2 (ACF of AR(1)). Let us use linear difference equations to compute the ACF for AR(1). For AR(1), φ₂ = 0 in (3.11) and we will get

ρ(h) = φρ(h − 1), h = 1, 2, . . . .

The characteristic polynomial is φ₁(z) = 1 − φz and its root is 1/φ. Since we have only one root, we use (3.14) to solve

ρ(h) = k(1/φ)^−h = kφ^h.

The initial value is ρ(0) = 1, and hence k = 1. Therefore the ACF for AR(1) is φ^|h|. We added absolute value so that ρ(h) = ρ(−h).

Let us now apply what have learned about linear difference equations to compute the ACF of AR(2).

ACF for AR(2) and beyond

Recall the linear difference equation for ρ(h) of an AR(2) process:

ρ(h) − φ₁ρ(h − 1) − φ₂ρ(h − 2) = 0, h = 1, 2, . . .

The characteristic polynomial is φ2(z) = 1−φ1z −φ2z²and let r1, r2be its two roots. Because the model is causal by assumption, we know that |r₁| > 1 and |r₂| > 1. The solution for this linear difference equation has 3 cases:

1. When r₁ and r₂ are real and distinct, then

ρ(h) = k1r₁^−h+ k2r^−h₂ . So ρ(h) → 0 exponentially as h → ∞.

2. When r₁ = r₂ = r are real and equal,

ρ(h) = r^−h(k₁+ k₂h).

So ρ(h) → 0 exponentially as h → ∞.

3. When r₁ and r₂ = r₁ are complex conjugate pair, then ρ(h) = k₁r₁^−h+ k₁r^−h₁ .

Here k₂ = k₁ because ρ(h) must be real. You can then simplify the above to a more interpretable form. Let us write the roots r₁ = re^iθ and r₁ = re^−iθ in polar coordinates with i the imaginary number and r = |r₁| = |r₂|. We also do so likewise for k₁ and k₁. Then it can be shown that (by Euler’s formula) the above reduces to

ρ(h) = ar^−hcos (hθ + b),

where a and b are constants determined by the initial values. So ρ(h) → 0 exponentially as h → ∞, but it does so in a sine wave/cyclical fashion. We call this ρ(h) the damped sine wave.

How about AR(p)? Consider the AR(p) process

X_t= φ₁X_t−1+ · · · + φ_pX_t−p+ Z_t.

The characteristic polynomial is φ_p(z) = 1 − φ₁z − · · · − φ_pz^p. Generalizing the linear difference equation in (3.11) to AR(p),

ρ(h) = φ₁ρ(h − 1) + φ₂ρ(h − 2) + · · · + φ_pρ(h − p), h = p, p + 1, . . . ,

with initial values for ρ(0), ρ(1), . . . , ρ(p − 1). Let r1, r2, . . . , rp be the distinct roots of φp(z).

Then the solution to this linear difference equation is

ρ(h) = k₁r₁^−h+ k₂r₂^−h+ · · · + k_pr_p^−h.

To get k₁, . . . , k_p, we substitute the initial values for ρ(0), ρ(1), . . . , ρ(p − 1) to the solution above, and we solve the resulting system of p simultaneous equations.

The ACF behaves the same as in AR(2), and we summarize their behavior as follows:

ˆ If the roots of the characteristic polynomial are real, the ACF decays to 0 exponentially.

So unlike MA(q) models, the ACF does not cut-off.

ˆ If there is a complex root, the ACF decays to 0 exponentially, but in a cosine-like oscillation (damped sine wave).

1 ar2 = a r i m a . sim (m o d e l=l i s t( ar =c(1.5 , -0.75) ) , n = 2 0 0 )

2

3 p l o t( ar2 , x l a b =" T i m e ") # l o o k s p e r i o d i c

4 a c f 2 ( ar2 , max. lag = 1 0 0 ) # s a m p l e ACF is c y c l i c a l

5

6 c o e f = c(1 , -1.5 , 0 . 7 5 ) # c o e f f i c i e n t s of the c h a r a c t e r i s t i c p o l y n o m i a l

7 p o l y r o o t(c o e f) # c o m p l e x c o n j u g a t e AR (2) r o o t s

8

9 arg = Arg(c o e f[ 1 ] )/(2*pi ) # arg in c y c l e s/u n i t t i m e ( f r e q u e n c y )

10 1/arg # p e r i o d

Remark 3.3. For AR(3) and above, we need to find the roots of cubic, quartic, quintic and higher degree characteristic polynomials. You can use Cardano’s method to solve cu-bic equations and Ferrari’s method for the quartic equation. Galois theory says that for quintic equations and above, there are in general no nice formulas involving radicals like the quadratic case, and the roots have to be computed numerically using root-finding algorithms such as Newton-Raphson method.