Complexity analysis and parallel implementation

This section presents a deep complexity analysis of the DFI method. In addition, the computational times of all the methods assessed in the previous subjective evaluation are presented.

Interpolation Algorithm analysis. The main part of our interpolation algorithm,i.e. steps 1 up to 3, operates only on integer data and does not use any multiplication. Operations are then restricted to bitshifts comparisons, and additions. Table 5.3 shows the number of each operation for each of the 5 steps. In addition, table5.3 shows the percentage of pixels that are processed by each step.

Number of operations per pixel +,- bitshifts comparisons % of processed pixels Step 1 0 0 0 25% Step 2 21 5 4 25% Step 3 21 5 4 50% Step 4 21 5 4 100% Step 5 10 2 4 100%

Mean operation number per pixel 46.75 10.75 11 100%

Table 5.3: Number of operations per pixel in the high resolution image for the proposed interpolation algorithm

Compared to other methods, the proposed DFI method exhibits a very low computational com- plexity. Moreover, the implementation uses integer data leading to faster computation time. For comparison purposes, the NEDI method [106] claims to perform 1300 multiplications per pixels.

Speed comparison. In order to illustrate its low complexity, our DFI method is compared to Im- ageMagick [37] interpolation functions and to the methods used in the subjective evaluation (section

5.1.9.2). Table5.4shows that the speed relative to the presented interpolation method is comparable to kernel based methods. This test shows that it is slightly faster that default ImageMagick filters but the DFI method achieves better visual results and lower memory consumption.

Interpolation algorithm (+ opening and saving) time for 250 iteration Peak memory

ImageMagick Lanczos filter 56.73 s 19404 ko

ImageMagick Mitchell filter 50.297 s 19384 ko

Proposed interpolation method 49.50 s 10504 ko

Table 5.4: Benchmark of interpolation methods, enlargement by 2 of a 512 by 512 image

For the subjective evaluation (section 5.1.9.2), the DFI method has comparable quality results with Photozoom [113] and pseudo LAD [165] methods. However, the DFI method is at least 50 times faster (see table5.5). As a consequence, the DFI method has the best tradeoff between quality and complexity of all the tested methods.

method execution time implementation

DFI method ≈ 0.06s compiled + multithreading

ImageMagick filters [37] ≈ 0.06s compiled + multithreading ICBI [61] ≈ 1.2s according to [61] compiled

iNEDI [7] >1000s Matlab

pseudo LAD [165] 3 to 4s compiled + multithreading

photozoom [113] 3 to 4s compiled + multithreading

Table 5.5: Execution times of the methods used in the subjective evaluation: 128 by 128 pixels image enlarged to a 512 by 512 pixels image

5.1.10.1 Speed enhancement solutions

The goal of the presented DFI is to achieve the higher quality with the minimum complexity. If the DFI has proved to be a fast method, its speed can be enhanced at a cost of a slight lost of quality. Different solutions have been envisaged.

First a speed enhancement solution taking advantage of the YCbCr color space characteristics can be designed. Since each channel of the YCbCr does not have the same visual importance the interpolation process should be different from the Y channel to the others. Indeed, the Y channel has to be processed with more care than the chrominance channels that can be processed by simpler and faster algorithm. In this case, the Y channel can be processed by our interpolation algorithm while the chrominance channels are processed by a simple bilinear filtering. A WPSNR decrease of 0.278dB is observed when compared to the version where all color channels are processed by the DFI method. However, the time gain between the two approaches is more than 50% while the loss of WPSNR is only 1%.

Secondly, a speed enhancement solution relies on an image activity classification. Indeed, for interpolating algorithms, the most critical results are located on edges and high energy or textured regions. On those areas, the interpolation method quality is of the most importance to get satisfying visual results. On the opposite side, smoother areas (low energy regions) of the image can be processed with less care for the same visual final result. Our idea is to choose the proper interpolation method for each pixel according to the pixel energy class. Since low energy pixels can be easily interpolated with a simpler and faster method, the overall complexity can be reduced compare to the case where all pixels are interpolated with our interpolation method. As an example, within the LAR framework, the quadtree structure gives us clues on the nature of the pixels. Typically, 2 by 2 pixels blocks represent areas of highest activity, such as texture or contour, being more difficult to interpolate. The proposed solution is then to process with the DFI method all the pixels represented by 2 by 2 blocks in the quadtree, while the rest of the image will be processed by a faster bilinear filtering. In addition, surrounding pixel of 2 by 2 blocks will also be processed by the DFI method to avoid frontier artifacts. Results show a negligible loss of WPSNR (0.0017 dB) while the speed gain is about 12%. The acceleration factor strongly depends on the ratio of classified pixels linked to the quadtree threshold TH and the nature of the image. On the 25 tested images, the speed gain

is ranging from 0% up to 80% with similar WPSNR losses.

In any cases, the Hybrid DFI algorithm leads to speed gain with a minimal quality loss. Classi- fication is here based on a quadtree decomposition of the image. However, any other classifier can be used, such as contour extraction in the low resolution image. Hybrid interpolation approach is therefore a general principle that can be applied on a wide application field.

For a maximal efficiency, both acceleration solutions can be combined. A speed increment of 59% with a slight quality loss of 0.277db is then observed. Figure5.11illustrates the visual effect of the hybrid solution. In particular, visually, the figures5.11a and5.11bare very similar.

Considering our speed performances, the DFI method is thus fast enough to be used for real time video interpolation. For example, it could be used to interpolate from SD to HD resolution. However, the DFI method is limited to enlargement ratio of 2n _{and the ratio between SD to HD is}

1.5. A solution exists to overcome this issue: it consists of adapting the DFI algorithm during the computation of missing pixels, as described in [32], by using a rectangular neighborhood of 1.5 ratio, instead of using a square neighborhood.

(a) Interpolated image, no acceler- ation

(b) Combined approach: YCbCr color space + Hybrid DFI

(c) Pixel classification for Hybrid DFI; white = DFI, black = bilinear interpolation

Figure 5.11: Combined approach: YCbCr color space + Hybrid DFI illustration, 2× enlargement