Up to now, we defined the notion of -REP for an ensemble of measurement matrices. This definition is what we call an “informational” characterization, in the sense that taking measurements by the ensemble potentially keeps more than 1− fraction of the information of the source. One can ask the natural question weather this has some “operational” implication, in the sense that after having the linear measurements, is it possible to recover the source up to an acceptable distortion (for example mean square error distortion)? Notice that, at the end of the day, it is the operational implication that matters because it practically deals with taking the measurements and reconstructing the signal via a recovery algorithm. More importantly, it takes into account the computational complexity of the recovery algorithm which is completely missing in the informational point of view. However, the informational point of view has its own advantages, namely, it allows to find the underlying fundamental limits of the problem without dealing with algorithmic issues. In this section, our goal is to briefly clarify these two different aspects for our proposed Hadamard construction.
Let us start form an example from polar codes for binary source compression which has lots of similarities with what we have studied in this section. As shown in [28], for a binary memoryless source with P(0) = p, for a large block length n, there is a matrix Gn of an approximate dimension nh2(p) × n such that the linear
measurement of the source by this matrix over F2 faithfully captures the randomness
of the source. This in its own only solves the encoding part of problem without directly addressing the decoding part, namely, it does not imply the existence of a decoder to recover the source from the measurements up to a negligible distortion (in this case error probability). Therefore, the operational picture is not complete yet. Fortunately, in the case of polar codes the successive cancellation decoder (or other decoders proposed in the literature) fills up the gap and shows that the informational characterization implies the operational one.
In the compressed sensing setting, the operational fundamental limits have been studied extensively for different measurement matrices and different recovery algorithms. In particular, for random Gaussian measurement matrices, the asymptotic sparsity-measurement rate behavior of different recovery algorithms, such as AMP and `1-minimization, has been vastly studied. We will mainly focus on the results
for the Gaussian matrices because, generally speaking, they give better measurement rates than other families of matrices. For the probabilistic model that we studied in this chapter, it has been observed that the required measurement rate of the Gaussian matrices is far from optimal. More precisely, for the sparsity δ very close to zero, the required measurement rate for successful recovery of the source under low-complexity algorithms such as `1-minimization and AMP scales like δ log2(1δ), which suffers from
an oversampling of order log2(1
δ) compared with the optimal measurement rate δ
4.6. Simulation Results 81
this extra factor by using spatially coupled Gaussian matrices and running AMP. This specifically shows that the operational limits meet the informational predictions. Recently, using extensive numerical simulations, it was shown that the optimal measurement rate δ still seems to be achievable by spatially coupling of partial Hadamard matrices, where the rows of the sub-matrices embedded in each block are selected completely at random from the rows of a Hadamard matrix [68]. Interestingly, it was observed that for this random construction, the resulting measurement rate of the Lasso and AMP recovery algorithms are comparable with (even slightly better than) that of random Gaussian matrices. It will be very interesting to find out if it is possible to derandomize this construction to obtain a deterministic partial Hadamard family of matrices with a close to optimal performance.
In the rest of this chapter, our goal is to operationally compare the performance of our proposed partial Hadamard matrices with that of the random Gaussian matrices. We will restrict ourselves to the dense Gaussian matrices without using the spatial coupling. To build our proposed measurement matrices, we use the REP criterion that we developed in this chapter. More precisely, we select those rows of the corresponding Hadamard matrix with a significant RID as we will explain further in Section 4.6. Since it is difficult to theoretically analyze the performance of the constructed matrices, we use numerical simulations to asses the performance for different signal models and different off-the-shelf algorithms from compressed sensing. Very briefly, the simulation results show that for our construction, the resulting measurement rate is comparable with (and even slightly better than) the random Gaussian matrices but it still suffers from the oversampling factor log2(1
δ) for small
sparsity values δ. This shows that even in our construction there is a gap between the informational and operational characterization and it seems that an extra spatial coupling as in [68] is still necessary to meet the optimal informational predictions.
4.6 Simulation Results
In this section, we asses the operational performance of the partial Hadamard matrices proposed in this chapter via numerical simulations. As explained in Section 4.5, this allows to numerically compare the gap that exists between the informational and operational characterizations.
4.6.1 Signal Model and the Recovery Algorithm
For simulations, we use a zero mean and unit variance sparse distribution
pX(x) = (1 − δ)δ0(x) + δpc(x), (4.7)
where δ0(x) is the unit delta measure at point zero, pc is the distribution of the
continuous part and δ ∈ {0.0, 0.1, . . . , 0.9, 1.0} is the RID of the signal. We use the mean square error (MSE) as distortion measure. The simulations are done with the Hadamard matrix of order N = 512. To build the measurement matrix A, we select those rows of HN with highest conditional RID, as stated in Section 4.4.3, until we
get an acceptable recovery distortion. One of the algorithms that we use to recover the signal is the `1-minimization algorithm:
ˆx(y) = arg min
where y = A x is the vector of measurements taken from the signal x. Another algorithm that we use is the AMP algorithm given by the following iteration:
zt= y − Aˆxt+1
γzt−1hη
0
t−1(A∗zt−1+ ˆxt−1)i, (4.9)
ˆxt+1= ηt(A∗zt+ ˆxt), (4.10)
where y = A x denotes the vector of linear measurements taken by A, γ is the mea- surement rate, han
1i =Pni=1ai/n, ηt(u) = (ηt,1(u1), . . . , ηt,N(uN)), where ηt,i(ui) =
E{X|ui = X + τtW}, with W ∼ N(0, 1) independent of the signal X and τt given
by the state evolution equation for AMP, is the soft-thresholding function designed for the known distribution of X. For initialization, we use ˆx0 = 0 and z0 = 0.
The behavior of the AMP algorithm was rigorously analyzed for random Gaussian matrices in [21]. Specifically, it was shown that the behavior of the AMP is fully characterized by a closed-form State Evolution equation.
4.6.2 Sensitivity to Signal Distribution
In this chapter, we proposed the Hadamard construction for a memoryless source with a given probability distribution. However, we showed that the polarization of the RID and as a result the matrix construction only depends on the RID of the source and not the detail of the distribution of the source. To assess how sensitive the construction is to the distribution of the signal, we do the simulations using three different distribution for the continuous part of signal distribution pc
in Equation (4.7). To recover the signal, we use `1-minimization algorithm as in
Equation (4.8). Figure 4.2 shows the boundary of the low-distortion region for the
`1-minimization algorithm where for the boundary we use 0.01 of the signal power as
the threshold. The results show that the required measurement rate is not sensitive to the distribution of the signal.
4.6.3 Comparison of the Performance of `1-minimization and AMP
In this part, we compare the performance of the two algorithms for a Bernoulli- Gaussian distribution in which pc is the normal distribution. Knowing the exact
distribution of the signal, we use MMSE soft-thresholding function for AMP as in Equation 4.10. Figure 4.3 shows the simulation results. Although AMP, with the thresholding function ηt designed for the known distribution of the signal, performs
slightly better than `1-minimization, there is still a gap compared with the optimal
line.
4.6.4 Comparison with Random Gaussian Matrices
In this section, we compare the performance of the Hadamard construction with the traditional random Gaussian matrices extensively used in compressed sensing.
The simulation results for `1-minimization are depicted in Figures 4.4 and 4.5. A
visual comparison shows that the Hadamard construction works slightly better than the Gaussian matrices, i.e., for a given measurement rate has less recovery distortion.
4.6. Simulation Results 83
Boundary of the Low Distortion Region for `1-minimization
Measuremen
t
Rate
Sparsity Factor (Rényi Information Dimension) 0 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 Gau Lap Uni Opt
Figure 4.2 – Boundary of the Low-Distortion Region for `1-minimization for Different
Signal Distributions. It is seen that the performance of the `1-minimization is not
sensitive to the distribution of the signal.
Boundary of the Low Distortion Region for AMP and `1-minimization
Measuremen
t
Rate
Sparsity Factor (Rényi Information Dimension) AMP `1-Min Opt 0.2 0.4 0.8 0.6 1 0.2 0.4 0.6 0.8 1
Figure 4.3 – Boundary of the Low-Distortion Region for AMP and `1-minimization.
The solid line shows the optimal boundary. Below this line no algorithm can work with a low distortion. As seen from the figure, AMP performs better than `1-minimization,
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.1 0.2 0.3 0.4 0.5 0.7 0.6 0.8 0.9 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Rate-Distortion Region for Hadamard Construction
Measurement Rate
Mean
Square
Error
(MSE)
Figure 4.4 – Rate-Distortion Region for Hadamard Construction and `1-minimization
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
Rate-Distortion Region for Gaussian Matrices
Mean Square Error (MSE) Measurement Rate 0.9 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Figure 4.5 – Rate-Distortion Region for Random Gaussian Matrices and `1-