We shall focus the discussions in this section exclusively on compressed sensing as the results on affine rank minimization is analogous. In Section 1.2, we have established the sharp RIP condition on different orders of RICs,
δkA<1/3 δtkA < r t−1 t for some t≥ 4 3,
for the recovery ofk-sparse signals in compressed sensing. For a general t >0, denote the sharp bound for δA
tk as δ∗(t). Then
δ∗(1) = 1/3 and δ∗(t) =
p
(t−1)/t, t ≥4/3.
A natural question is: What is the value ofδ∗(t) fort <4/3 and t6= 1? That is, what
is the sharp bound for δtkA when t < 4/3 and t 6= 1? We have the following partial answer to the question.
Proposition 1.4.1. Let y =Aβ where β ∈ Rp is k-sparse. Suppose 0 < t < 1 and
tk ≥0 to be an integer
When tk is even and δAtk < 4−tt, the`1 minimization (1.2)withB ={0}recovers
β exactly.
When tk is odd andδA
tk <
√
t2−1/k2
4−2t+√t2−1/k2, the`1 minimization(1.2)withB ={0}
recovers β exactly.
In addition, the following result shows that δ∗(t) ≤ 4−tt for all 0 < t < 4/3. In
particular, whent= 1, the upper boundt/(4−t) coincides with the true sharp bound 1/3.
0 0.5 1 1.5 2 2.5 3 3.5 4 0 0.2 0.4 0.6 0.8 1 t δ *
Figure 1.2: Plot of δ∗ as a function of t. The dotted line is t = 4/3.
Proposition 1.4.2. For0< t <4/3, >0and any integerk ≥1,δtkA < 4−tt+is not
sufficient for the exact recovery. Specifically, there exists a matrix A with δA
tk = 4−tt
and a k-sparse vector β0 such that βˆ =6 β0, where βˆ is the minimizer of (1.2) with
B={0}.
Propositions 1.4.1 and 1.4.2 together show that δ∗(t) = 4−tt when tk is even and
0< t <1. We are not able to provide a complete answer for δ∗(t) when 0< t <4/3.
We conjecture thatδ∗(t) = 4−tt for all 0< t <4/3. Figure 1.2 plotsδ∗(t) as a function
of t based on this conjecture for the interval (0,4/3).
Our results show that exact recovery of k-sparse signals in the noiseless case is guaranteed ifδAtk <p(t−1)/tfor somet≥4/3. It is then natural to ask the question: Among all these RIP conditionsδtkA < δ∗(t), which one is easiest to be satisfied? There
is no general answer to this question as no condition is strictly weaker or stronger than the others. It is however interesting to consider special random measurement matrices A= (Aij)n×p where Aij ∼ N(0,1/n), Aij ∼ 1/√n w.p.1/2 −1/√n w.p.1/2 , or Aij ∼ p 3/n w.p.1/6 0 w.p.1/2 −p 3/n w.p.1/6 .
concentration of measure. For these random measurement matrices, Theorem 5.2 of Baraniuk et al. (2008) shows that for positive integer m < n and 0< λ <1,
P(δmA < λ)≥1−2 12ep mλ m exp −n(λ2/16−λ3/48). (1.33) Hence, for t≥4/3,
P(δtkA <p(t−1)/t)≥1−2 exptklog(12e/pt(t−1)) + log(p/k)
−n t−1 16t − (t−1)3/2 48t3/2 .
For 0< t <4/3, using the conjectured value δ∗(t) = 4−tt, we have
P(δtkA < t/(4−t))≥1−2 exptk(log(12(4−t)e/t2) + log(p/k))
−n t2 16(4−t)2 − t3 48(4−t)3 .
It is easy to see whenp, k,andp/k→ ∞, the lower bound ofnto ensureδAtk < t/(4−t) orδA
tk <
p
(t−1)/t to hold in high probability isn ≥klog(p/k)n∗(t), where
n∗ , t/16(4t2−t)2 − t3 48(4−t)3 t <4/3; t/t16−1t − (t48−1)t3/32/2 , t≥4/3.
For the plot ofn∗(t), see Figure 1.3. n∗(t) has minimum 83.2 whent= 1.85. Moreover, among integer t, t= 2 can also provide a near-optimal minimum: n∗(2) = 83.7.
We should note that the above analysis is based on the bound given in (1.33) which itself can be possibly improved.
0 1 2 3 4 5 6 7 0 100 200 300 400 500 600 t n *
2
ROP: Matrix Recovery via Rank-one Projections
2.1
Introduction
Accurate recovery of low-rank matrices has a wide range of applications, including quantum state tomography (Alquier et al. 2013, Gross et al. 2010), face recognition (Basri and Jacobs 2003, Cand`es et al. 2011), recommender systems (Koren et al., 2009), and linear system identification and control (Recht et al., 2010). For example, a key step in reconstructing the quantum states in low-rank quantum tomography is the estimation of a low-rank matrix based on Pauli measurements (Gross et al. 2010, Wang 2013). And phase retrieval, a problem which arises in a range of signal and image processing applications including X-ray crystallography, astronomical imaging, and diffraction imaging, can be reformulated as a low-rank matrix recovery problem Cand`es et al. (2013); Cand`es et al. (2011). See Recht et al. (2010) and Cand`es and Plan (2011) for further references and discussions.
Motivated by these applications, low-rank matrix estimation based on a small number of measurements has drawn much recent attention in several fields, includ- ing statistics, electrical engineering, applied mathematics, and computer science. For example, Cand`es and Recht (2009), Cand`es and Tao (2010) and Recht Recht (2011) considered the exact recovery of a low-rank matrix based on a subset of uniformly
sampled entries. Negahban and Wainwright (2011) investigated matrix completion under a row/column weighted random sampling scheme. Recht et al. (2010), Cand`es and Plan (2011), and Cai and Zhang (2013b,a, 2014b) studied matrix recovery based on a small number of linear measurements in the framework of restricted isometry property (RIP), and Koltchinskii et al. (2011) proposed the penalized nuclear norm minimization method and derived a general sharp oracle inequality under the condi- tion of restrict isometry in expectation.
The basic model for low-rank matrix recovery can be written as
y=X(A) +z, (2.1)
where X : Rp1×p2 →
Rn is a linear map, A∈Rp1×p2 is an unknown low-rank matrix,
and z is a noise vector. The goal is to recover the low-rank matrix A based on the measurements (X, y). The linear map X can be equivalently specified by n p1 ×p2
measurement matrices X1,· · · , Xn with
X(A) = (hX1, Ai,hX2, Ai,· · · ,hXn, Ai)|, (2.2)
where the inner product of two matrices of the same dimensions is defined ashX, Yi=
P
i,jXijYij. SincehX, Yi= Trace(X|Y), (2.1) is also known as trace regression.
A common approach to low-rank matrix recovery is the constrained nuclear norm minimization method which estimates A by
ˆ
A= arg min
M
{kMk∗ :y− X(M)∈ Z}. (2.3)
Here kXk∗ is the nuclear norm of the matrix X which is defined to be the sum of
example, Z = {0} in the noiseless case and Z is the feasible set of the error vector
z in the case of bounded noise. This constrained nuclear norm minimization method has been well studied. See, for example, (Recht et al. 2010, Cand`es and Plan 2011, Oymak and Hassibi 2010, Cai and Zhang 2013b,a, 2014b).
Two random design models for low-rank matrix recovery have been particularly well studied in the literature. One is the so-called “Gaussian ensemble” (Recht et al. 2010, Cand`es and Plan 2011), where the measurement matrices X1,· · · , Xn are ran-
dom matrices with i.i.d. Gaussian entries. By exploiting the low-dimensional struc- ture, the number of linear measurements can be far smaller than the number of entries in the matrix to ensure stable recovery. It has been shown that a matrix A of rank r
can be stably recovered by nuclear norm minimization with high probability, provided that n &r(p1+p2) (Cand`es and Plan, 2011). One major disadvantage of the Gaus-
sian ensemble design is that it requires O(np1p2) bytes of storage space for X, which
can be excessively large for the recovery of large matrices. For example, at least 45 TB of space is need to store the measurement matricesMi in order to ensure accurate
reconstruction of 10000×10000 matrices of rank 10. (See more discussion in Section 2.5.) Another popular design is the “matrix completion” model (Cand`es and Recht 2009, Cand`es and Tao 2010, Recht 2011), under which the individual entries of the matrix A are observed at randomly selected positions. In terms of the measurement matrices Xi in (2.2), this can be interpreted as
X(A) = (hei1e | j1, Ai,hei2e | j2, Ai,· · · ,heine | jn, Ai) | (2.4) where ei = (0,· · · ,0, ith z}|{
1 , 0,· · · ,0) is the ith standard basis vector, and i1,· · · , in
and j1,· · · , jn are randomly and uniformly drawn with replacement from{1,· · · , p1}
Recht (2011), additional structural assumptions, which are not intuitive and difficult to check, on the unknown matrix A are needed in order to ensure stable recovery under the matrix completion model. For example, it is impossible to recover spiked matrices under the matrix completion model. This can be easily seen from a simple example where the matrix A has only one non-zero row. In this case, although the matrix is only of rank one, it is not recoverable under the matrix completion model unless all the elements on the non-zero row are observed.
In this chapter we introduce a“Rank-One Projection” (ROP) model for low-rank matrix recovery and propose a constrained nuclear norm minimization method for this model. Under the ROP model, we observe
yi = (β(i))|Aγ(i)+zi, i= 1, ..., n (2.5)
where β(i) and γ(i) are random vectors with entries independently drawn from some
distributionP, andziare random errors. In terms of the linear map X :Rp1×p2 →Rn
in (2.1), it can be defined as
[X(A)]i = (β(i))|Aγ(i), i= 1,· · · , n. (2.6)
Since the measurement matrices Xi = β(i)(γ(i))| are of rank-one, we call the model
(2.5) a “Rank-One Projection” (ROP) model. It is easy to see that the storage for the measurement vectors in the ROP model (2.5) is O(n(p1 +p2)) bytes which is
significantly smaller than O(np1p2) bytes required for the Gaussian ensemble.
We first establish a sufficient identifiability condition in Section 2.2 by considering the problem of exact recovery of low-rank matrices in the noiseless case. It is shown that, with high probability, ROP withn &r(p1+p2) random projections is sufficient
minimization. The required number of measurements O(r(p1+p2)) is rate optimal
for any linear measurement model since a rank rmatrix A∈Rp1+p2 has the degree of
freedom r(p1+p2−r). The Gaussian noise case is of particular interest in statistics.
We propose a new constrained nuclear norm minimization estimator and investigate its theoretical and numerical properties in the Gaussian noise case. Both upper and lower bounds for the estimation accuracy under the Frobenius norm loss are obtained. The estimator is shown to be rate-optimal when the number of rank-one projections satisfies either n & (p1 +p2) log(p1 +p2) or n ∼ r(p1 +p2). The lower bound also shows that if the number of measurements n < rmax(p1, p2), then no estimator
can recover rank-r matrices consistently. The general case where the matrix A is only approximately low-rank is also considered. The results show that the proposed estimator is adaptive to the rankrand robust against small perturbations. Extensions to the sub-Gaussian design and sub-Gaussian noise distribution are also considered.
The ROP model can be further simplified by taking β(i) = γ(i) if the low-rank matrix A is known to be symmetric. This is the case in many applications, includ- ing low-dimensional Euclidean embedding (Trosset 2000, Recht et al. 2010), phase retrieval (Cand`es et al. 2013, Cand`es et al. 2011), and covariance matrix estimation (Chen et al. 2013, Cai et al. 2013b,a). In such a setting, the ROP design can be simplified to symmetric rank-one projections (SROP)
[X(A)]i = (β(i))|Aβ(i).
We will show that the results for the general ROP model continue to hold for the SROP model whenAis known to be symmetric. Recovery of symmetric positive def- inite matrices in the noiseless and `1-bounded noise settings has also been considered
in a recent paper by Chen et al. (2013) which was posted on arXiv at the time of the writing of the present chapter. Their results and techniques for symmetric positive
definite matrices are not applicable to the recovery of general low-rank matrices. See Section 2.6 for more discussions.
The techniques and main results developed in the chapter also have implications to other related statistical problems. In particular, the results imply that it is possible to accurately estimate a spiked covariance matrix based only on one-dimensional projections. Spiked covariance matrix model has been well studied in the context of principal component analysis (PCA) based on i.i.d. data where one observes p- dimensional vectorsX(1),· · · , X(n) iid∼ N(0,Σ) with Σ = Ip+Σ0 and Σ0being low-rank
(Johnstone 2001, Birnbaum et al. 2013, Cai et al. 2013b,a). This covariance structure and its variations have been used in many applications including signal processing, financial econometrics, chemometrics, and population genetics. See, for example, Fan et al. (2008), Nadler (2010), Patterson and Reich (2006), Price et al. (2006), Wax and Kailath (1985). Suppose that the random vectors X(1),· · · , X(n) are not directly observable. Instead, we observe only one-dimensional random projections of X(i),
ξi =hβ(i), X(i)i, i= 1, ..., n,
where β(i)iid∼ N(0, Ip). It is somewhat surprising that it is still possible to accurately
estimate the spiked covariance matrix Σ based only on the one-dimensional projec- tions {ξi : i = 1, ..., n}. This covariance matrix recovery problem is also related to
the recent literature on covariance sketching (Dasarathy et al., 2012, 2013), which aims to recover a symmetric matrix A (or a general rectangular matrixB) from low- dimensional projections of the form X|AX (or X|BY). See Section 2.4 for further
discussions.
The proposed methods can be efficiently implemented via convex programming. A simulation study is carried out to investigate the numerical performance of the proposed nuclear norm minimization estimators. The numerical results indicate that
ROP with n≥5rmax(p1, p2) random projections is sufficient to ensure the exact re-
covery of rank r matrices through constrained nuclear norm minimization and show that the procedure is robust against small perturbations, which confirm the theo- retical results developed in the chapter. The proposed estimator outperforms two other alternative procedures numerically in the noisy case. In addition, the proposed method is illustrated through an image compression example.
The rest of this chapter is organized as follows. In Section 2.2, after introducing basic notation and definitions, we consider exact recovery of low-rank matrices in the noiseless case and establish a sufficient identifiability condition. A constrained nuclear norm minimization estimator is introduced for the Gaussian noise case. Both upper and lower bounds are obtained for estimation under the Frobenius norm loss. Section 2.3 considers extensions to sub-Gaussian design and sub-Gaussian noise distributions. An application to estimation of spiked covariance matrices based on one-dimensional projections is discussed in detail in Section 2.4. Section 2.5 investigates the numerical performance of the proposed procedure through a simulation study and an image compression example. A brief discussion is given in Section 2.6. The main results are proved in the Appendix (Chapter A.2).