PLAMEN KOEV
Abstract. In the following survey we look at structured matrices with what is referred to as low displacement rank. Matrices like Cauchy, Vandermonde, Polynomial Vandermonde, Chebyshev Vandermonde, Toeplitz, Hankel and others only depend onO(n) parameters instead of n2. This suggests that linear systems of these types should be solvable with some degree of effort less thanO(n3). The same should also extend to theLU-factorization and inversion. Also the inverses of (say) Vandermonde matrices do not have Van-dermonde structure, yet they should have similar properties when it comes to solving linear equations. The property that describes the above structured matrices and their inverses (and Schur complements) is that they have a low displacement rank. Exploiting the displacement structure of a matrix allows us to obtainO(n2) algorithms for solvingAx=b, obtaining theLU-factorization and for inversion of matrices with low displacement rank. The present survey does not contain any new results and is entirely based on the excellent papers by Vadim Olshevsky and Thomas Kailath noted in the references. Our task was to provide an outline of the main results for matrices with low displace-ment rank and provide the reader with an insight into the underlying logic of this theory.
Let the matricesF, A∈Cn×n be given. LetR∈Cn×n be a matrix satisfying a
Sylvester type equation
∆F,A(R) =F·R−R·A=G·B
for some rectangular matrices G∈Cn×α, B∈Cα×n, where the numberαis small
in comparison ton.
The pair of matricesG, B above is referred to as the{F, A}-generator of Rand the smallest possible inner sizeα among all {F, A}-generators is called a {F, A}-displacement rank ofR. This is the so-called Toeplitz-like displacement operator.
The Hankel-like displacement operator is defined as: ∆F,A(R) =R−F·R·A=G·B.
Basic classes of Structured Matrices
Toeplitz-like: F =Z1 A=Z−1
Toeplitz-plus-Hankel-like: F =Y00 A=Y11
Cauchy-like: F = diag(c1, . . . , cn) A= diag(d1, . . . , dn)
Vandermonde-like: F = diag(1 x1, . . . , 1 xn) A=Z1 Chebyshev-Vandermonde-like: F = diag(x1, . . . , xn) A=Yγ,δ Date: July, 1999. 1
Here Zφ = 0 0 · · · 0 φ 1 0 · · · 0 0 1 . .. ... .. . . .. . .. . .. ... 0 · · · 0 1 0 , Yγ,δ = γ 1 0 · · · 0 1 0 1 . .. ... 0 1 . .. . .. 0 .. . . .. . .. 0 1 0 · · · 0 1 δ ,
i.e.,Zφ is the lower shiftφ-circulant matrix andYγ,δ=Z0+Z0T +γe1eT1 +δe1eT1.
Example 1. Cauchy Matrix:
c1 c2 . .. cn · 1 c1−d1 1 c1−d2 · · · 1 c1−dn 1 c2−d1 1 c2−d2 · · · 1 c2−dn .. . ... . .. ... 1 cn−d1 1 cn−d2 · · · 1 cn−dn − 1 c1−d1 1 c1−d2 · · · 1 c1−dn 1 c2−d1 1 c2−d2 · · · 1 c2−dn .. . ... . .. ... 1 cn−d1 1 cn−d2 · · · 1 cn−dn · d1 d2 . .. dn = 1 1 · · · 1 1 1 · · · 1 .. . ... . .. ... 1 1 · · · 1 .
Therefore the displacement rank of a Cauchy matrix is one.
The Displacement Structure is Inherited During Inversion
If F R−RA = GB then AR−1−R−1F = −(R−1G)(BR−1), so R−1 has a
similar displacement structure and the same{A, F}displacement rank as the{F, A} displacement rank ofR.
Similarity Transformations Preserve the Displacement Rank
IfF R−RA=GB andR1=T1−1RT2then
F1R1−R1A1=G1B1,
where F1 =T1−1F T1, A1=T2−1AT2, G1 =T1−1G, and B1 =BT2. This allows us
to transform a structured matrix from one class to another.
The Displacement Structure is Inherited During Schur Complementation
Lemma 1. Let the matrix
R= d 1 u1 l1 R (1) 22
satisfy Sylvester type displacement equation: ∆F1,A1(R1) = f1 0 ∗ F2 ·R1−R1· a1 ∗ 0 A2 =G1·B1, whereG1∈Cn×α andB ∈Cα×n.
Ifd16= 0then the Schur complement R2=R (1) 22 −
l1u1
d1 satisfies the displacement
equation F2·R2−R2·A2=G2·B2, where 0 G2 =G1− 1 1 d1l1 ·g1, 0 B2 =B1−b1· 1 d1 1u1 ,
andg1 andb1 are the first row ofG1 and the first column ofB1, respectively.
Proof. From the standard Schur complementation formula R1= 1 0 1 d1l1 I · d1 0 0 R2 · 1 d1 1u1 0 I , we get f1 0 ∗ F2 · d1 0 0 R2 − d1 0 0 R2 · a1 ∗ 0 A2 = 1 0 −1 d1l1 I ·B1·G1· 1 −1 d1u1 0 I . Equating the (2,2) block entries one obtains the desired result.
Note: The requirement thatF andAbe lower and upper triangular is essential. Otherwise the above (and the Fast Algorithm) doesn’t work. What this means is that if the (1,1) entry ofF andAis 1 then one step of Gaussian Elimination must leaveF andAT unchanged.
Fast Gaussian Elimination for a Structured Matrix
• Recover from the generator the first column and the first row of
R1= d 1 u1 l1 R (1) 22 .
Note: This must take O(1) flops per entry or the cost of the algorithm goes beyondO(n2).
• Now one has the first column
1
1 d1l1
ofLand the first row
d1 u1
ofU in theLU factorization ofR1.
• Compute a generator of the Schur complement ofR1using 0 G2 =G1− 1 1 d1l1 ·g1, 0 B2 =B1−b1· 1 d1 1u1 , whereg1 andb1 are the first row ofG1 and the first column ofB1,
Example 2. Consider the Sylvester type equation for a Cauchy-like matrix: 4 0 0 0 5 0 0 0 6 · 1/3 2/2 3 1/4 3/3 4/2 1/5 4/4 5/3 − 1/3 2/2 3 1/4 3/3 4/2 1/5 4/4 5/3 · 1 0 0 0 2 0 0 0 3 = 1 0 1 1 1 2 · 1 2 3 0 1 1
The first column of Land the first row ofU in the LU decomposition are:
R=LU = 1 0 0 3/4 ∗ 0 3/5 ∗ ∗ · 1/3 1 3 0 ∗ ∗ 0 0 ∗ .
The generators of the Schur complement are:
0 G2 = 1 0 1 1 1 2 − 1 3/4 3/5 · 1 0 = 0 0 1/4 1 2/5 2 , therefore G2= 1/4 1 2/5 2 . Also, 0 B2 = 1 2 3 0 1 1 − 1 0 · 1 3 9 = 0 −1 −6 0 1 1 , therefore B2= −1 −6 1 1 .
So the Schur complementR2 satisfies the following displacement equation: 5 0 0 6 ·R2−R2· 2 0 0 3 = 1/4 1 2/5 2 · −1 −6 1 1 Therefore R2= 1/4 −1/4 2/5 −2/15 . (If diag(ci)·R−R·diag(di) =G·B thenrij =
gi·bj
ci−dj, wheregi andbj are theith row ofGand the jth column ofB, respectively.)
We continue the same way.
The second column ofLand the second row of U in theLU decomposition are:
R=LU = 1 0 0 3/4 1 0 3/5 8/5 ∗ · 1/3 1 3 0 1/4 −1/4 0 0 ∗ .
The generators of the Schur complement are:
0 G3 = 1/4 1 2/5 2 − 1 8/5 · 1/4 1 = 0 0 0 2/5 , so G3= 0 2/5 .
Also 0 B3 = −1 −6 1 1 − −1 1 · 1 −1 = 0 −7 0 2 , so B3= −7 2 .
The Schur complementR3 satisfies the displacement equation 6 ·R3−R3· 3 = 0 2/5 · −7 2 = 4/5 Therefore R3= 4/15
and the LU decomposition is
R=LU = 1 0 0 3/4 1 0 3/5 8/5 1 · 1/3 1 3 0 1/4 −1/4 0 0 4/15 .
Pivoting for Matrices with Displacement Structure
Partial pivoting may be applied to matrices with displacement structure that satisfy the displacement equation
F1R1−R1A1=G1B1
withF1-diagonal matrix. After the row interchange, the matrix ˆR1=P1R1satisfies
the same displacement equation with the diagonal matrix F1 replaced by another
diagonal matrix ˆF =P1F1P1T, and withG1replaced by ˆG1=P1G1. Now
F1R1−R1A1=G1B1
implies
(P1F1P1T)(P1R1)−(P1R1)A1= (P1G1)B1.
Fast GEPP Algorithm for Structured Matrices
• Recover from the generator the first column of
R1= d 1 u1 l1 R (1) 22 .
Note: This will depend on the form of the matrices F1 and A1. The
procedure was specified for a Cauchy matrix. Procedures exist for the recovery of the matrixR1from its displacement equation for all basic classes
of matrices with displacement structure.
• Next determine the position (say) (k,1) of the entry with maximal mag-nitude in the first column. Let P1 be a permutation of the first and the
k-th entries. Interchange the first and the k-th diagonal entries ofF1 and
interchange the first and thek-th rows in the matrix G1.
• Then recover the first row of P1R1 from the generator. Now one has the
first column 1 1 d1l1
ofLand the first row
d1 u1
• Compute a generator of the Schur complement ofR2of P1R1 using 0 G2 = G1− 1 1 d1l1 ·g1, 0 B2 = B1−b1· 1 d1 1u1 ,
whereg1 andb1 are the first row ofG1 and the first column ofB1,
respec-tively.
• Proceeding recursively, one finally obtains factorization R1=P LU,
whereP =P1· · ·Pn−1 andPk is the permutation used at the k-th step of
the recursion.
Fast Inversion for Matrices with Displacement Structure
For the next paragraph we will assume that we know how to solve Rx =b in O(n2) operations for the nonsingular matrixR that satisfiesF R−RA=GB. To
do this we can either use Fast GE or first transform R into a Cauchy-like matrix (we will see later how) and use Fast GEPP.
FromF R−RA=GB we obtain
AR−1−R−1F=−(R−1G)(BR−1),
thus R−1 satisfies a very similar displacement equation. If we know the {A, F}
generators{−R−1G, BR−1}ofR−1then we can recoverR−1from this displacement
equation inO(n2) time. Note that algorithms exist for the recovery of the matrix
R from the displacement equation F R−RA = GB for most classes (actually all famous classes, Toeplitz, Chebyshev-Vandermonde, etc) of matrices with low displacement rank in O(1) operations per entry, i.e. in O(n2) operations for the
entire matrix.
We can compute −R−1G and BR−1 in O(n2) time as follows. First compute
R−1G by solving α times the system Rx = gi, i = 1,2, . . . , α, where gi are the
columns of G and α is the displacement rank of G. Since solving Rx = b takes O(n2) time and α is small in comparison with n we can obtain R−1G in O(n2) time.
Then computeBR−1 in the following way: BR−1= (R−TBT)T.
R−TBT is the solution of αequations RTx=b
i, i= 1,2, . . . , α. Each of those
equations can be solved inO(n2) time because ifR=LU thenRT =UTLT. If the
matrixRwas first transformed into another type of structured matrix (say Cauchy-like from Toeplitz-Cauchy-like in order to apply GEPP) then we haveT1−1RTT2=UTLT.
We can still solve Rx=b in O(n2) time because as we will see later the matrices
T1 and T2 will be diagonal matrices, Fast Trigonometric Transforms or products
thereof.
Having obtained the generators ofR−1 inO(n2) time we can recover the matrix
R−1 from the generators and the displacement equation inO(n2) time. The total
time required for inversion ofR isO(n2).
Transformation of Toeplitz-like matrices into Cauchy-like matrices
As described earlier, we need to be able to convert the other classes of structured matrices into Cauchy-like matrices before we can apply partial pivoting.
If R is a Toeplitz matrix then Z1R−RZ1 = GB where the rank of G is not
greater than 2. Matrices that satisfyZ1R−RZ1=GB whereGis of low rank are
referred to as Toeplitz-like matrices.
Here is how the Toeplitz-like matrices are transformed into Cauchy-like matrices. Consider the (normalized) Discrete Fourier Transform matrix
F = 1 √ ne 2πi n (k−1)(j−1) 1≤k,j≤n , and the matrices
D1 = diag(1, e 2πi n , . . . , e 2πi n (n−1)), D−1 = diag(e πi n, e 3πi n , . . . , e (2n−1)πi n ), D0 = diag(1, e πi n, . . . , e (n−1)πi n ).
The following factorizations are well known: Z1=F∗D1F, Z−1=D−01F
∗D
−1F D0.
Substituting the above intoZ1R−RZ−1=GB one obtains
D1(F RD−01F∗)−(F RD −1
0 F∗)D−1= (F G)(BD∗0F∗),
i.e.,F RD−01F∗ is a Cauchy-like matrix.
After applying the Fast GEPP we obtain F RD0−1F∗ = P LU, so we get the factorization R = F∗P LU F D0−1. Solving Rx = b will require O(n2) operations
and will consist of the application of two (normalized) DFTs, one diagonal scaling, a permutation, a forward and backward substitution for a total ofO(n2) operations.
Transformation of Vandermonde-like into Cauchy-like matrices
The Vandermonde matrixV =hxji−1i
1≤i,j≤nsatisfies the displacement equation
D1 xV −V Z T 1 = 1 x1 1 x2 . . . 1 xn T · 1 0 . . . 0 . By analogy we shall refer to any matrix R with low{D1
x, Z T 1}-displacement rank as a Vandermonde-like matrix. IfD1 xV −V Z T 1 =GB, thenRF∗ is a Cauchy-like matrix: D1 x(RF ∗)−(RF∗)D∗ 1=G(BF ∗). Toeplitz-plus-Hankel-like matrices
LetF =Y00,A=Y11,T be a Toeplitz matrix andH be a Hankel matrix:
T = t0 t1 · · · tn−1 t−1 t0 · · · tn−2 · · · . .. · · · t−n+1 t−n+2 · · · t0 , H = t0 t1 · · · tn−1 t1 t0 · · · tn · · · . .. · · · tn−1 tn · · · t2n−2 . We have rank (Y00(T+H)−(T+H)Y11)≤4
Matrices with low {Y00, Y11}-displacement rank are referred to as
The matrix Yγδ with γ, δ ∈ {−1,1} or γ =δ= 0 can be diagonalized by Fast
Trigonometric Transform Matrices.
Y00=SDSS, Y11=CDCCT, where C = "r 2 nqjcos (2k−1)(j−1)π 2n # 1≤k,j≤n and S = "r 2 n+ 1sin kjπ n+ 1 # 1≤k,j≤n
are the (normalized) Discrete Cosine Tansform-II and Discrete Sine Transform-I matrices, respectively, (q1=√12, q2=. . .=qn= 1), and
DC = 2·diag 1,cos π n, . . . ,cos (n−1)π n , DS = 2·diag 1,cos π n+ 1, . . . ,cos nπ n+ 1 .
IfRis a Toeplitz-plus-Hankel-like Matrix, thenSRC is a Cauchy-like matrix: The equationY00R−RY11=GB yields
DS(SRC)−(SRC)DC= (SG)(BC).
Chebyshev-Vandermonde Matrices
LetT0, T1, . . .andU0, U1, . . .be the Chebyshev Polynomials of the first and the
second kind respectively. For nonzerox1, . . . , xn the matrices
VT = T0(x1) T1(x1) · · · Tn−1(x1) T0(x2) T1(x2) · · · Tn−1(x2) .. . ... . .. ... T0(xn) T1(xn) · · · Tn−1(xn) , and VU = U0(x1) U1(x1) · · · Un−1(x1) U0(x2) U1(x2) · · · Un−1(x2) .. . ... . .. ... U0(xn) U1(xn) · · · Un−1(xn)
are referred to as Chebyshev-Vandermondematrices. The Chebyshev polynomials satisfy the relations
T0(x) = 1, T1(x) =x, Tn(x) = 2xTn−1(x)−Tn−2(x) U0(x) = 1, U1(x) = 2x, Un(x) = 2xUn−1(x)−Un−2(x) Consider F =D1 x = diag 1 x1 , 1 x2 , . . . , 1 xn
and A = W ≡ 0 2 0 −2 0 · · · 0 0 2 0 −2 · · · . .. . .. . .. . .. . .. . .. . .. . .. ... .. . . .. . .. . .. −2 . .. . .. 0 . .. 2 0 · · · 0 = 2 [n 2] X i=1 (−1)i−1·(Z0T)2i−1,
where Z0 as above is the lower circular shift. Let D0 = diag(12,1, . . . ,1). The
Chebyshev-Vandermonde matrices satisfy D1 x ·(VT(x)D0)−(VT(x)D0)·W = 1 x1 1 x2 .. . 1 xn · 1 2 0 −1 0 1 0 −1 . . . , D1 x·VU(x)−VU·W = 1 x1 1 x2 .. . 1 xn · 1 2 0 −1 0 1 0 −1 . . . .
By analogy we will refer to matrices with small {D1
x, W}-displacement rank as Chebyshev-Vandermonde-like.
Alternatively, one can prove that the Chebyshev-Vandermonde-like matrices have low {2Dx, Y11},{2Dx, Y00} or {2Dx, Z1+Z1T}-displacement rank. All
dis-placement operators above describe in fact the same class of matrices. A matrix that has a low rank with respect to one displacement operator will have a low dis-placement rank with respect to the other operators (but not necessarily the same). If a matrix R has a low {2Dx, Y11},{2Dx, Y00} or {2Dx, Z1+Z1T}-displacement
rank then RC, RS or RF∗ respectively are Cauchy-like matrices, where S, C and
F are appropriate Discrete Trigonometric Transforms described earlier in the text.
References
[1] I. Gohberg, T. Kailath, and V. Olshevsky,Fast Gaussian elimination with partial pivoting for matrices with displacement structure, Math. Comp., 64 (1995), pp. 1557–1576.
[2] T. Kailath and V. Olshevsky,Displacement structure approach to Chebyshev-Vandermonde and related matrices, Integral Equations Operator Theory, 22 (1995), pp. 65–92.
[3] T. Kailath and V. Olshevsky,Displacement-structure approach to polynomial Vandermonde and related matrices, Linear Algebra Appl., 261 (1997), pp. 49–90.
