Algorithm 3.2: Consistent Cycle Spinning (CCS) for Signal Denoising input: data u = Ty, initial guessbx0,
quadratic penaltyτ > 0, noise variance σ2,
transform matrix T, and scalar shrinkageT .
set: t← 1 and µ0← 0 repeat b wt = T (u + τTbxt−1+ µt−1)/(1 + τ); Kσ2/(1 + τ) bxt= T†wbt− µt−1/τ µt= µt−1− τ( bwt− Tbxt) t← t + 1
until stopping criterion return bxt
whereτ > 0 is the penalty parameter and µ ∈ RK N is the vector of Lagrange multipli- ers. The condition w = Tx asserted by the penalty function constrains w to the column space of T, which is equivalent to the consistency condition w = TT†w. Although the
quadratic penalty term inL does not influence the final solution, it typically improves the convergence behaviour of the iterative optimization [50]. To solve the minimization, we alternate between solving the problem for w for x fixed and vice versa, which corresponds to the alternating direction method of multipliers (ADMM) algorithm [22]. Implementa- tion of CCS is summarized in Algorithm 3.2.
3.6
Numerical Evaluation
We now present some numerical results illustrating the concepts discussed in this section. We divide experiments into two main parts. In the first, we illustrate the results of Theo- rem 3.1 by considering general linear inverse problems. In the second part, we highlight the MMSE denoising capability of CCS by considering a statistical signal model based on Lévy processes. Advantage of using the Lévy signal model is that we can then compare the results to the MMSE estimator computed by the message-passing algorithm developed in Chapter 4.
3.6.1 Convergence of Cycle Spinning
We illustrate Theorem 3.1 with two simple examples. In the first example, we consider the estimation of a piecewise constant signal of length N = 128 corrupted by AWGN corresponding to an input signal-to-noise ratio (SNR) of 30 dB. An interesting property of such signals is that they can be sparsified with the finite-difference operator, which justifies the use of TV regularization [51]. Since the TV regularizer corresponds to an
ℓ1-penalty applied to the finite differences of the signal, our theorem indicates that it can
also be minimized with cycle spinning when W corresponds to the Haar-wavelet basis with one level of decomposition and a zero weight in the lowpass. In Figure 3.1, we plot the per-iteration gapCTV(bxt) − CTV∗
, wherebxt is computed with cycle spinning andC TV
is the TV-regularized least-squares cost. We setλ = 0.05 and, following our analysis, we
set the step-size toγt = 1/(4pt). As expected, we observe that, as t→ ∞, we have that
CTV(bxt) − CTV∗
→ 0. Moreover, we note that, for large t, the slope ofCTV(bxt) − CTV∗
in log-log domain tends to −1/2, which indicates the asymptotic rate of convergence O (1/pt).
100 101 102 103 104 105 10−4 10−2 100 102 100 10−4 10−2 100 102 t CTV (bx t) − C ∗ TV
Figure 3.1: Estimation of a sparse signal from noisy measurements. We plot the gap
CTV(bxt) − CTV∗
against the iteration number t. The plot illustrates the convergence of cycle spinning to the minimizer of the cost functionCTV.
(a) (b) (c)
(d) (e)
Figure 3.2: Reconstruction of Cameraman from blurry and noisy measurements. (a) orig- inal, (b) blurry, (c) standard wavelet-based ISTA (SNR = 19.10 dB), (d) reconstruction with TV (SNR = 21.58 dB), (e) wavelet-based reconstruction with cycle spinning (SNR = 21.58 dB).
In the second example, we consider an image-deblurring problem where the Cameraman image of size 256× 256 is blurred with a 7 × 7 Gaussian kernel of standard deviation 2 with the addition of AWGN of varianceσ2= 10−5. In Figure 3.2, we present the result of
the reconstruction with three different methods: standard Haar-domainℓ1-regularization, anisotropic TV [52], and cycle spinning with 1D Haar-basis functions applied horizontally and vertically to imitate TV. The regularization parameters for the standard wavelet ap- proach and TV were optimized for the least-error performance. The regularization param- eter of cycle spinning was set by rescaling the regularization parameter of TV according to
λCS=p2KλTV, with K = 44. Therefore, we expect cycle spinning to again match the TV
3.6. Numerical Evaluation 100−1 100 101 9 Brownian Motion AWGN σ2 ∆ SN R [ d B] CCS−MAP/MMSE frame−MAP/MMSE ortho−MAP/MMSE ortho−ST
Figure 3.3: SNR improvement as a function of the level of noise for Brownian mo- tion. The wavelet-denoising methods by reverse order of performance are: standard soft-thresholding (ortho-ST), optimal shrinkage in a wavelet basis (ortho-MAP/MMSE), shrinkage in a redundant system (frame-MAP/MMSE), and optimal shrinkage with con- sistent cycle spinning (CCS-MAP/MMSE).
100−1 100 101 7 Laplace Motion AWGN σ2 ∆ SN R [ d B] CCS−MMSE CCS−MAP frame−MAP frame−MMSE ortho−MMSE ortho−MAP
Figure 3.4: SNR improvement as a function of the level of noise for a Lévy process with Laplace-distributed increments. The wavelet-denoising methods by reverse order of per- formance are: ortho-MAP (equivalent to soft-thresholding with fixedλ), ortho-MMSE,
frame-MMSE, frame-MAP, CCS-MAP, and CCS-MMSE. The results of CCS-MMSE are com- patible with the ones of the reference MMSE estimator in Chapter 4.
solution. It is clear from Figure 3.2 that cycle spinning outperforms the standard wavelet regularization (improvement of at least 2 dB). As expected, the solution obtained by cycle spinning exactly matches that of TV both visually and in terms of SNR.
3.6.2 Statistical Estimation with CCS
We experimentally evaluate CCS by considering two versions of the algorithm: CCS-MAP and CCS-MMSE that use MAP and MMSE shrinkage functions, respectively. We test the estimation performance of the algorithms on Lévy processes that will be discussed with much greater detail in Chapter 4. One can view Lévy processes as continuous-time analogs of random walks. Their defining property is that they have independent and stationary increments [53, 54], which implies that the applications of finite-difference operator on samples of a Lévy process decouples it into a sequence of independent random variables. In the experiments, we consider the denoising of four types of Lévy processes of lengths
N =2048 in the Haar-wavelet domain. All the results were obtained by averaging 10 ran-
10−1 100 101 0 10 Compound Poisson AWGN σ2 ∆ SN R [ d B] CCS−MMSE frame−MMSE ortho−MMSE ortho−ST
Figure 3.5: SNR improvement as a function of the level of noise for a compound Poisson process (piecewise-constant signal). The wavelet-denoising methods by reverse order of performance are: ortho-ST, ortho-MMSE, frame-MMSE, and CCS-MMSE. The results of CCS-MMSE are compatible with the ones of the reference MMSE estimator obtained using message passing in Chapter 4.
100−1 100 101 4 Levy Flight AWGN σ2 ∆ SN R [ d B] CCS−MMSE CCS−MAP frame−MAP frame−MMSE ortho−MMSE ortho−MAP
Figure 3.6: SNR improvement as a function of the level of noise for a Lévy flight with Cauchy-distributed increments. The wavelet-denoising methods by reverse order of per- formance are: ortho-MAP, ortho-MMSE, frame-MMSE, frame-MAP, CCS-MAP, and CCS- MMSE. The results of CCS-MMSE are compatible with the ones of the reference MMSE estimator obtained using message passing in Chapter 4.
dom realizations of the problem for various noise levels. The metric used for comparisons is the SNR improvement ∆SNR ¬ 10 log10 kx − yk2 2 kx − bxk2 2 (3.16) The four statistical distributions for our signals are (see Figure 4.2 for an illustration of their realizations)
– Gaussian: This is the best-known example of a Lévy process. We set the distribution
of increments toN (0, 1).
– Laplace: We set the scale parameter to one.
– Compound-Poisson: We consider the sparse scenarios and set the mass probability
at 0 to 0.6. The size of the jumps follow the standard Gaussian distribution.
– Cauchy: We set the distribution of the increments to be Cauchy (α = 1) with scale
parameterρ = 1.
The wavelet denoising methods compared are the standard orthogonal wavelet soft-thresholding (ortho-ST), statistically optimal shrinkages in the orthogonal wavelet domain (ortho-