Methods using Local Structure Similarity

Internal image denoising with a single image is popular and usually has a low computational load. Earlier techniques focused on recovering noisy pixels from their neighboring noisy pixels e.g. Gaussian filtering, bilateral filtering, and total

variation. Later algorithms focused on re-occurrence of patches [Glasner et al., 2009] in the noisy image to reconstruct the noise-free image, examples are non- local means [Buades et al., 2005], BM3D [Dabov et al., 2007b], WNNM [Gu et al., 2014], SAIST [Dong et al., 2013], SAPCA [Dabov et al., 2009], and TSID [Zhang et al., 2010]. These algorithms are effective for areas with repetitive texture, however, on the downside, they suffer when they attempt to find corresponding matches for in- frequent patches i.e. the patches that are rarely present in the image. To overcome this issue, some methods [Lou et al., 2009; Yan et al., 2012] proposed alternatives al- beit with limited applicability. Moreover, when the noise is strong, internal denoising performance degrades drastically as it struggles to find correct reference patches.

State-of-the-art techniques in internal image denoising [Buades et al., 2005; Dabov et al., 2007b, 2009; Lebrun et al., 2013; Dong et al., 2013; Foi et al., 2007; Zhang et al., 2010] exploit repetitive local patterns that frequently occur in natural images, by selecting and grouping similar patches for denoising.

2.2.2.1 Non-local Means

The non-local means (NLM) algorithm takes benefit of the self-similarity property or redundancy in natural images. The self-similarity property states that for each patch in an image there are similar patches. This assumption is valid where the patches are in the neighborhood or within few pixels from the center of the reference patch. Hence, the redundancy in this scenario is termed as local means. Making use of this property, non-local means [Buades et al., 2005; Goossens et al., 2008] denoise images by computing a weighted average of non-local similar patches, with the weights set as the Euclidean distances between their pixel values. [Efros and Leung, 1999] have practiced the self-similarity property for texture image synthesis.

2.2.2.2 Block Matching and Three Dimensional Filtering

Block matching and three dimensional (3D) filtering is abbreviated as BM3D. Collab- orative filtering with block matching and its variants [Dabov et al., 2007b; Foi et al., 2007; Dabov et al., 2009] are prominent baseline methods that exploit patch similarity in a 2D transform domain.

The core idea lies in the imposition of structural similarity among patches in each group by analyzing the subspace of the transform coefficients. The denoising of individual patches relies on the implicit assumption that insignificant coefficients correspond to the noise component and thus can be truncated via thresholding or attenuated via Wiener filtering.

Figure 2.2: Example of grouping patches from noisy images. Each image shows a reference patch “R” (in red) and similar looking patches (in blue).

Figure 2.3: BM3D framework. The process is repeated as indicated by the dashed lines.

any regularization. BM3D searches for patches similar to a reference patch in a local neighborhood as shown in Figure 2.2 and groups them into a three dimensional (3D) block, hence, the term block-matching. The three-dimensional block is denoised, and the resulting patch is inserted into its original location. The process is repeated for every patch in the image. This process of grouping the patches, is different than clustering, as each patch can be part of multiple blocks, as opposed to clustering where each patch can only belong to one cluster.

The BM3D algorithm is a two-stage process as shown in Figure 2.3. The dis- tinction separating the stages is how search is performed for similar patches and which procedure is employed for shrinking coefficients. In the first stage, the similar patches are searched in the noisy image, and then hard thresholding is applied to the 3D blocki.e. the values below a certain threshold are set to zero. The outcome of this stage is a denoised image as each patch is re-inserted into their original location. In the second stage, the similar patches are searched in the resulting denoised image from the first stage. The reason for seeking patches again from the first denoised image is because it is more reliable to to search similar patches using the denoised image of first step than in the noisy image itself. However, two blocks are formed,

one block consist of noisy patches while other containing patches from the denoised image of the first step. The shrinkage procedure for the second stage is Wiener fil- tering. The weights of noisy patches are used to approximate the block of denoised patches. In other words, the purpose of the first step is to achieve a denoised image to provide reliable similar-patches while the second phase is to get the final denoised image.

2.2.2.3 Non-local Bayes

Similar to BM3D, Non-Local Bayes (NLB) [Lebrun et al., 2013] consists of a two- step process that estimates the values of a latent patch from the mean and covariance matrix of its similar patches. To obtain an invertible covariance matrix NLB algorithm keeps a fixed number of similar patches which are more than the size of the noisy patch. NLB is also a two-step process like BM3D but having both steps identical with only using the estimate from the first step. NLB results are better in case of color images than its counterparts.

2.2.2.4 Weighted Nuclear Norm Minimization

Nuclear Norm Minimization (NNM) is becoming popular in recent years due to its ability to efficiently recover low-rank matrices. NNM uses frobenius-norm to observe the difference between the noisy image y and the latent image x. NNM has an analytical solution and can be solved by soft-thresholding of singular values. In case of NNM, soft-thresholding is performed using single threshold value. However, this assumption is not reasonable as different singular values have different importance and therefore, should be treated differently. [Gu et al., 2014] proposed weighted nuclear norm minimization (WNNM) where every single value is thresholded based on their relative importance based on non-local self-similarity.

2.2.2.5 Principal Component Analysis

Principal Component Analysis (PCA) is successfully employed in many other image restoration fields, and it also found its way to image denoising. Initially, [Muresan and Parks, 2003] proposed the principal component analysis (PCA) on sliding image patches. [Zhang et al., 2010] proposed local grouping of the similar pixels in the neighborhood and then apply PCA to remove the noise, this process is repeated by further refining the output of the first stage. [Deledalle et al., 2011] presented the comparison between different patch-based PCA effect on image denoising.

Figure 2.4: A general framework for the external denoising methods.

2.2.2.6 Dual Domain & Progressive Image Denoising

Recently, new algorithms are proposed which operates at pixel level rather than the patches-level. Example of such algorithm is Dual Domain image denoising (DDID) [Knaus and Zwicker, 2013]. It is a simple algorithm which iterates between the spatial domain and the frequency domain using bilateral filtering and short time Fourier transform, respectively. DDID is a guided method which uses the intermedi- ate denoised image as an input to the final stage, although all the stages are identical except the intermediate image. The results are comparative to BM3D besides its sim- plicity. Furthermore, Progressive image denoising (PID) [Knaus and Zwicker, 2014] is a variant of DDID and improved by incorporating arbitrary fine time steps in it- erations. Similar to DDID, it also performs denoising using kernels in two domains: spatial domain and frequency domain. The visual performance of PID is better than DDID.