Introduction to convolution and correlation
5.3 Description of convolution
k=0
x[k]δ[n − k].
(5.5) More generally, the bilateral version of Equation (5.4) is given by
x[n] =
∞ k=−∞
x[k]δ[n − k]. (5.6)
Equations (5.5) and (5.6) must be true, since the function δ[n − k] is only ever equal to 1 when n = k, otherwise it is zero.
5.3 Description of convolution
5.3.1 Using linear superposition
In Chapter 2 we saw how, if a linear system produced y1[n] in response to x1[n], and y2[n] in response to x2[n], then if we add the inputs first and then apply them, the system produces a correspondingly summed version of the two original out-puts. Moreover, if the inputs are weighted, then the outputs will be weighted by the same amount. These manifestations of linear systems we termed the properties
of superposition and proportionality, which are succinctly encapsulated in the relationship written as
∞ k=0
akxk[n] → h[n] →∞
k=0
akyk[n], (5.7)
where h[n] represents the linear system (i.e. its impulse response). Now we are getting to the heart of the matter of convolution, for it involves three principles:
• Any input signal may be described as a series of weighted, shifted impulse functions.
• Linear systems are superpositional.
• Linear systems are proportional.
As an example, imagine we had a simple linear system whose impulse response h[n] was
h[0] = 4, h[1] = 2, h[2] = 1, h[3] = 0.5. (5.8)
We want to use convolution to predict its output in response to the input signal x[k]
given by:
x[0] = 2, x[1] = 1, x[2] = −1, x[3] = −2, x[4] = 3.
(5.9) Each signal value x[k] is a weighted, shifted impulse function. Thus the system’s response when k = 0 (i.e. when x[0] = 2) will be 8, 4, 2, 1, that is, this is an impulse response weighted by 2. When k = 1, the system’s response to x[1] will be 4, 2, 1, 0.5.
However, this response will occur one time interval later, since k has advanced one unit. Since the system is still responding to the first value, the overall response to the inputs x[0] and x[1] is obtained by summing the individual responses, with a delay applied to the second response. This process is repeated for each of the input signal values. Grouping the individual responses together and shifting appropriately, we obtain when summed:
8 4 2 1
4 2 1 0.5
−4 −2 −1 −0.5
−8 −4 −2 −1
12 6 3 1.5
y[n] = 8 8 0 −8 7.5 3.5 2 1.5 n = 0, . . . , 7
In this example we clearly see how impulse representation, superposition and proportionality are applied in the mechanics of the convolution process.
5.3.2 Using time reversal of the impulse response
It is worth noting that discrete-time convolution may also be described by the equation
y[n] =
4 k=0
x[k]h[n − k]. (5.10)
If you compare this equation with Equation (5.3), you will notice that the positions of x[k] and h[n] have been switched. This does not matter, because the principle of commutativity (which we also examined in Chapter 2) states that the order of operations in a linear system may be rearranged without affecting the outcome. In this particular case, if we were to swap the position of x[k] and h[n] in Equation (5.10), we would write
y[n] =
3 k=0
h[k]x[n − k], (5.11)
since k ranges from 0 to 3. In general, however, we can state that
∞ k=0
x[k]h[n − k] =
∞ k=0
h[k]x[n − k]. (5.12)
Whether we use Equations (5.3) or (5.10) makes no difference, since in the first case we are time-reversing the signal and in the second we are time reversing the impulse response. Similarly, we can describe the mechanics of convolution in a manner that is slightly different to that of the above scheme. It involves time-reversal of the impulse response, with its final value aligned with the first value of the signal, that is, x[0], and the rest of the impulse response extending back in time. We now multiply each value of the impulse response with its corresponding value of the signal. When we have performed this for all values of the impulse response, we sum the products, and the result of this sum represents one new value of the output signal y[n]. Next, the entire time reversed version of the impulse response is shifted one place to the right, that is, delayed in time by one sampling interval, and the product summation is repeated to obtain the next value of y[n], and so on. Using the above values for x[n] and h[k], to calculate y[0], the initial position of the signals x[n] and h[−k] would be:
x[n] = (0) (0) (0) 2 1 −1 −2 3
h[−k] = 0.5 1 2 4
This gives y[0] = 0.5 × 0 + 1 × 0 + 2 × 0 + 4 × 2 = 8.
Note the first three values of x[n] are in parentheses, since the values for x[−3]
to x[−1] have not been defined. By convention, they are said to be zero. To obtain
Figure 5.3 Screenshot of convolve.exe y[1], the signal positions are
x[n] = (0) (0) (0) 2 1 −1 −2 3
h[−k] = 0.5 1 2 4
This gives y[0] = 0.5 × 0 + 1 × 0 + 2 × 2 + 4 × 1 = 8.
If we continue this operation, then we will obtain the same sequence as before, that is, 8, 8, 0,−8, 7.5, 3.5, 2, 1.5. This confirms that the convolution operation may be effected by sliding the time-reversed impulse response through the input signal and accumulating the products at each time interval to obtain one new output point.
If you run the program convolve.exe, located on the CD that accompanies this book, in the folder Applications for Chapter 5\Convolution\, you can see a demonstration of the mathematical equivalence of these two representations of the convolution sum, as defined by Equation (5.12). A screenshot of the program is shown in Figure 5.3.
Using this program, you can enter any desired values for h[k] and x[n] and compute their convolution, using either the button labelled Convolve x[k]h[n − k] or the button labelled Convolve h[k]x[n − k]. For example, if you choose the impulse response and input signal as given by the sequence in Equations (5.8) and (5.9), both buttons will execute code that computes the output sequence 8, 8, 0,−8, 7.5, 3.5, 2, 1.5. Listings 5.1 and 5.2 show code fragments for each button, that is, code that performs convolution using the left- and right-hand side of Equation (5.12), respectively.
for n:=0 to M1+N1-2 do begin
y[n]:=0;
for k:=0 to N1-1 do begin
y[n]:=y[n]+x[k]*h[n-k];
end;
end;
Listing 5.1
for n:=0 to M1+N1-2 do begin
y[n]:=0;
for k:=0 to M1-1 do begin
y[n]:=y[n]+h[k]*x[n-k];
end;
end;
Listing 5.2
The main bulk of both of these code fragments is taken up, at the beginning, with placing the data from the string grids into the arrays for holding the impulse response and input signal, that is, h[M1] and x[N1]. These arrays are defined from−100 to 100, because we want to avoid problems with negative indices during the convolution process. In both code fragments, the arrays continue to be filled until an empty cell is detected in the string grid. These input details are omitted from the listings, but can of course be seen in the source code, which is available on the CD in the same directory as the program. After this point, we enter the loops where the convolution is performed.
Remember, two loops are always required for convolution – one to index n; this determines which value of y[n] is being computed, and one to index k; this controls the product summations of h[n] and x[n]. In the sixth line from the bottom of these code fragments appears the very core of the matter, the convolution sum.
It is important to recognise that if we have an impulse response M points in length, and input signal N points in length, then the output sequence will always be M + N − 1 points in length. This is acknowledged in the code fragments by the n index loop, which loops from n= 0 to M1 + N1 − 2.
5.3.3 Fourier interpretation of convolution
At this stage, we do not want to go into too much detail about how we understand convolution from a frequency perspective, because we have not covered the Fourier transform in detail yet; that comes in Chapters 6 and 7. However, we will just state here that convolution in one domain is equivalent to multiplication in the other; in
0 1 2 3 4 5 6 7 8
Figure 5.4 The relationship between convolution and multiplication in the time and frequency domain. (a) Two identical signals, x[n] and h[n]; (b) spec-trum of the two signals; (c) the convolution of x[n] and h[n], that is, y[n]; (d) spectrum of y[n]. Analysis of parts (b) and (d) confirms Equation (5.13)
fact, the precise relationship is given by
y[n] = h[n] ∗ x[n] = NF−1{H [k]X[k]}, (5.13)
where N is the length of the Fourier transform. We can demonstrate this quite eas-ily by choosing both x[n] and h[n] to comprise four pulses and four zeros, that is, 1, 1, 1, 1, 0, 0, 0, 0. (We zero-pad the sequence for reasons associated with com-puting the fast Fourier transform (FFT), something we shall look at in Chapter 6.) Figure 5.4(a) shows the signal x[n] (or h[n]) and Figure 5.4(b) depicts its discrete Fourier transform. When we convolve these two sequences, we get the signal y[n], as shown in Figure 5.4(c), whose Fourier transform is shown in Figure 5.4(d), Y [k].
If you examine the numerical values of the various harmonics, you indeed see that Equation (5.13) is true. For example, the first harmonic for both H [k] and X[k], as shown in Figure 5.4(b), has the value 0.5. The equivalent harmonic for the multiplied spectra should therefore be 8× 0.5 × 0.5 = 2. This relationship holds for all other harmonics in the spectra.
5.3.4 Simple filtering using convolution
In Chapter 4, we used both differential and difference equations to obtain the impulse responses of simple analog filters. If we express these impulse responses in discrete
Figure 5.5 Screenshot of the program convolfilt.exe
form, we can apply them in conjunction with convolution to filter discrete signals;
these signals we can either design ourselves on the computer, or we can obtain them from real sensors using a suitable acquisition system. The point here, of course, is that we can exploit a digital methodology to replicate the behaviour of analog signals and systems. Now, digital filter design can be a complicated affair, so will explore this subject in detail in Chapters 11 and 12. However, to demonstrate in a simple way how convolution may be used for filtering, try running the program convolfilt.exe, which may be found in the folder Applications for Chapter 5\Convolfilter\, on the CD that accompanies this book; Figure 5.5 shows a screenshot of this program.
This program generates a square wave of user-defined frequency, within the range 1–100 Hz (the default is 10 Hz). It also generates the impulse response of a first order low-pass RC filter, of the kind we examined in the previous chapter, whose impulse response, you may remember, is given by
h(t) = 1
RC e−t/RC= 1
Ae−t/A (5.14)
and whose−3 dB frequency point is given by 1/(2πA). After generating both the square wave signal and the impulse response, the code automatically uses it to filter the signal, employing the convolution sum directly as given by Equation (5.3). The three code fragments given in Listing 5.3 show, respectively, the calculation of the impulse response, the input square wave and the convolution sum.
fc:=scrollbar1.Position;
edit1.Text:=inttostr(fc);
a:=1/(2*pi*fc);
edit2.Text:=floattostr(a);
for n:=0 to siglen do begin
t:=n/siglen;
h[n]:=(1/(a*normalise))*exp(-t/a);
end;
. . i:=1;
freq:=scrollbar2.position;
edit3.Text:=inttostr(freq);
ms:=round(siglen/(freq*2));
for n:=-siglen to siglen-1 do begin
if (n mod ms=0) then i:=i*-1;
x[n]:=i;
end;
. .
for n:=0 to siglen-1 do begin
y[n]:=0;
for k:=0 to siglen-1 do begin
y[n]:=y[n]+h[k]*x[n-k];
end;
end;
Listing 5.3
In Listing 5.3, the cut-off frequency of the filter is calculated using the position of the scroll bar 1, and the frequency of the square wave is taken from the position of scroll bar 2.
For a given frequency of signal, you can see that as the cut-off frequency of the filter falls (i.e. as its time constant increases), the square wave progressively loses its edge definition, and furthermore, it falls in magnitude. The rounding of the edge occurs because these are rich in high frequency components. As the filter cut-off point falls, more and more of these high frequency components are lost, that is, the overall energy of the signal is reduced, which causes this fall in the absolute magnitude. This is illustrated in Figure 5.6. Here, we are dealing with a 5 Hz square wave, shown in part (a). In Figure 5.6(b), we see the signal filtered by a filter with a cut-off frequency
c Time, s
Figure 5.6 (a) A 5 Hz square wave; (b) after filtering with a 50 Hz cut-off RC filter;
(c) after filtering with a 3 Hz cut-off RC filter
of 50 Hz. The output signal exhibits a little distortion, with slight rounding of the rising and falling edges. In contrast, Figure 5.6(c) depicts the same square wave after passing through a filter with a cut-off frequency of 3 Hz. It has lost much of its original shape and is severely attenuated.
Once again, we stress that although this is a very simple example, it is the prin-ciple of the matter that is so significant. If we had built this filter out of physical components, that is, a resistor and a capacitor, and fed it with a suitable square wave from an electronic function generator, then, provided we had used component values and a signal frequency that were consistent with those of our program, we should expect to see the same output signal on an oscilloscope screen as the one that appears in the program. This takes us back to the ideas we proposed at the start of this chapter: if we know the impulse response of a linear system, then, by using it with the convolution sum, we can predict its response to any input signal. Here we have used an electronic filter, but linear systems are everywhere in engineering and nature; as long as we have an impulse response, or we can express it in terms of the system’s component parts, then we can replicate its performance within a DSP environment.
5.3.5 Efficient convolution algorithms
Computationally, convolution is an intensive process, at least in its direct-form imple-mentation. In general, if the impulse response and input signal comprise M and N
values, respectively, then it will require M × N multiplications, additions and shift operations to achieve the convolution process, that is, to generate the output signal.
The combined operations of multiplication and addition, which invariably occur together with convolution, are often denoted by the phrase multiply-accumulate, or MAC, for short. By shifting, we mean the sliding or shifting of the impulse response through the signal, as described above. Now this M × N product may not seem unduly excessive until you think about the number of signal values in a typical digital audio signal. In general, CD quality audio is sampled at 44.1 kHz.
For a stereo recording, this means that a 1 s signal comprises 88,200 sample points.
If we wished to filter the signal with an impulse response of say 3000 points (not uncommon), then the number of operations required is 264,600,000. Not surpris-ingly therefore, DSP practitioners are always looking for ways to circumvent the direct form implementation, or at least to speed up its execution. There are four principle ways to reduce the time of calculation, and all of these we shall explore in this book.
The first, and most obvious way, is to make dedicated hardware that excels at fast, repetitive arithmetic. These digital signal processors, or DSP chips as they are called, are used almost exclusively for real-time processing, and their architecture is optimised for that all-important MAC operation. They are blindingly fast – building systems that incorporate them and programming them can be a real joy; more on these in Chapter 14.
The second way is to express the impulse response as an infinite impulse response (IIR). In general, this is only possible if the impulse response conforms to a straight-forward analytical expression, that is, it can be written as a mathematical function.
This is certainly the case for the first order filter we have just seen, so we can expect to see more of this method later in the book. IIR filters are exceptionally efficient, in some cases reducing the number of MACs required by several orders of magnitude. In the program Convolfilt.exe, the convolution uses the impulse response directly, which is composed of 1000 signal points. The signal also comprises 1000 values, so 106 MACs are performed to get the output signal. If we had adopted an IIR approach, as we will in Chapter 12, the same process could have been achieved with approximately 103MACs!
The third method of improving the efficiency is to use a Fourier approach, that is, we take both the impulse response and the signal into the Fourier domain, multiply them, take the result and pass it through an inverse Fourier transform, which yields the output signal y[n]. This certainly reduces the time required, since fewer arithmetic operations are necessary to achieve the same result. However, it is algo-rithmically complex and only works if we obey certain stipulations; more on this in Chapter 7.
The fourth and final way of reducing the time taken only works if the impulse response is symmetrical about some centre value. If it has an odd number of points and the values conform to the relationship
h(k) = h(M − k − 1), k = 0, 1, . . . ,(M − 1)
2 , (5.15)
then the convolution sum may be recast as
y[n] =
((M−1)/2)−1
k=0
h[k]{x[n − k] + x[n − (M − 1 − k)]}. (5.16) A similar expression exists for symmetrical filters comprising an even number of values (Ifeachor and Jervis, 1993). It should be emphasised, however, that whilst Equation (5.16) is certainly useful for general off-line processing using a personal computer, DSP devices are often designed for executing Equation (5.3) directly, in which case the mathematically more efficient realisation would confer no speed advantage.