Objective and subjective evaluation of HDR video compression algorithms

Despite the extensive research that has been conducted into development and evaluation of QA/VQA metrics for both LDR and HDR content, little has been done to evaluate existing HDR video compression algorithms (as outlined previously in Section 3.3) using both QA metrics and subjective experiments. Koz et al. [KD12] conducted a comparative survey on HDR video compression which compares the two different (non-backward and backward compatible) approaches to HDR video compression as explained earlier in Section 3.1. However, this work has a few shortcomings. First, it does not bring together objective and subjective evaluation techniques to provide a comprehensive evaluation. Second, it focuses on the two approaches to HDR video compression thereby largely ignoring the evaluation of individual algorithms across a large set of sequences.

Recently, Hanhart et al. [HRE15] conducted an evaluation of nine HDR video com- pression algorithms submitted in response to MPEG CfE [LFH15] to evaluate the feasibility of supporting HDR and WCG content using the HEVC [SOHW12] codec. The paper con-

cludes that the proposals submitted to MPEG can noticeably improve the standard HDR video coding technology and QA metrics such as PSNR-DE1000, HDR-VDP-2 and PSNR- Lx can reliably detect visible difference. However, this work has a few shortcomings. First, the reference sequences are not uncompressed source sequences and are stored as 12-bit non-linearly quantized RGB signal representation. Second, the psychophysical evaluation uses the same training samples as the test samples. Third, the naive participants were in- structed to find some specific errors in the video sequences. These issues can result in biased subjective opinions.

Azimi et al. [ABO∗_{15] conducted an objective and subjective evaluation study to}

compare the compression efficiency of two possible HDR video encoding schemes i.e the PQ algorithm and tone mapping-inverse tone mapping with metadata. The objective eval- uation was conducted using four QA metrics and the subjective evaluation was conducted using 18 participants. Results demonstrate the accuracy and monotonicity indexes of the four QA metrics and concludes that the video quality predicted by HDR-VDP and VIF has the highest correlation with subjective results. The correlation was computed using statisti- cal non-parametric tests such Spearman’s Rho Rank correlation. Furthermore, it concludes that for specific bitrates, HDR video generated by the PQ scheme were rated higher than the videos reconstructed using the inverse tone-mapping scheme.

Dehkrodi et al. [BDAPN14] conducted a similar evaluation which focuses on the compression efficiency of the HEVC codec compared to the state-of-the-art H.264/AVC codec. The authors use four HDR video sequences and convert them using the PQ algo- rithm. The converted sequences are then encoded using both the HEVC and H.264/AVC and correspondingly decoded and evaluated using several objective QA metrics. The out- put sequences are also subjectively evaluated by a rating based experiment using 17 par- ticipants. Results suggested that the sequences encoded using the HEVC codec outper- forms their H264/AVC counterparts by _≈ 10.18% in terms of quality and yet achieves bitrate savings of approximately_≈25.08%. Similar evaluations have been conducted by Dong et al. [DNP12], Rerabek et al. [RHKE15], Hanhart et al. [HKE∗_{15], Narwaria et}

al. [NPDSLC15].

It is to be noted however, that although the works mentioned here present some interesting results, it does not compare the state-of-the-art published or patented algorithms which were proposed before the MPEG CfE call and mostly focuses on the proprietary algorithms presented to MPEG. Furthermore, the works mentioned uses too few HDR video sequences to conclusively draw any generic conclusion. With the growing interest on HDR image/video compression algorithms, it is imperative that a comprehensive objective and subjective evaluation the pre-MPEG algorithms with a robust methodology for evaluation would ideally set the benchmark following which other compression algorithms can be comprehensively evaluation. Moreover, a deep understanding of HDR video compression on the whole can be obtained by such an evaluation. Chapter 6 of this thesis describes such

a work which has been conducted in order to comprehensively evaluate six published and patented video compression algorithms against a large set of HDR video sequences.

4.6

Summary

In this chapter, the reader has been provided a brief overview of both objective and sub- jective image quality evaluation techniques. The objective evaluation techniques discussed in this chapter include a brief overview of several objective QA metrics representing en- ergy difference, structural and perceptual QA metrics. Later in Chapters 6 and 7, these QA metrics have been used extensively for HDR video compression evaluation purposes. Ad- ditionally, this chapter also provides an overview of the previous research conducted on the design and evaluation of LDR/HDR QA metrics, subjective evaluation of TMOs and finally objective and subjective evaluation of HDR video compression algorithms.

In the next chapter, the reader will be introduced to a novel research work which has been conducted to evaluate the viewer’s perspective and choice of HDR video over LDR video given certain viewing conditions. The work presented in the next chapter is the first step to answer the research question discussed previously in Chapter 1.