Overview of Chapters - High Performance Multiview Video Coding

This dissertation consists of four chapters. Chapter 2 presents the design of two novel parallel motion estimation and disparity estimation algorithms to significantly reduce encoding time. Both algorithms are applicable to AVC/H.264 multiview extension and HEVC/H.265 multiview extension. Chapter 3 discusses the use of GOP level parallelism in conjunction with an efficient scheduling scheme to achieve efficient parallel encoding on computer cluster. Chapter 4 discusses high level optimization techniques to reduce the evaluation for multiple functional units and proposes ME/DE algorithms tailored for HEVC/H.265. Chapter 5 concludes this dissertation by summarizing the key points for this work.

Chapter 2 Massively Parallel Motion and

Disparity Estimation Algorithms

1

Multiview video coding (MVC) has recently received considerable attention. It is proposed as an extension of H.264/AVC standard for multiple video source compres- sion. To resolve the extremely high computational complexity of MVC (and in fact other advanced video coding techniques), requires development of suitable parallel algorithms that are amenable to implementation on low cost massively parallel architecture (MPA); platforms that have found a common place due to recent advances in the parallel computer architecture. The high complexity of MVC is due to its prediction structure, where motion estimation (ME) between the frames, and disparity estimation (DE) between the views, contribute to more than 99% of overall complexity of the coder. This chapter presents the development and implementation of a scalable massively parallel fast search algorithm to, significantly, reduce the

and sub-optimal fast search algorithms. The proposed massively parallel fast search algorithm (DZfast), when evaluated over eight views, outperforms the existing full search and fast search MVC algorithms by a factor up to 245.8 and 8.4, respectively. This speedup comes at no or minute loss in rate-distortion (RD) performance.

2.1 Introduction

Multiview video coding (MVC) is defined in Annex H of the state-of-the-art video coding standard H.264/AVC [3]. Common application of MVC are 3D movies/TV, free view TV, and immersive teleconferencing [25], [26]. In these applications, multiple cameras are deployed to capture dynamic scenes simultaneously. MVC in addition to taking the advantage of inter-frame temporal similarity in motion estimation (ME), as in conventional video coding techniques, exploits inter-view similarity to achieve about twice the higher coding efficiency compared with coding each view independently (simulcast). Inter-view similarities are used in disparity estimation (DE) in between the neighboring view video sequences. The most common technique for ME/DE is block matching [25]. This technique computes motion vector (MV) and disparity vector (DV) between two best matched blocks of pixels in two frames (temporal or inter-view). Block matching, even in single view video coding, is a time-consuming process amounting to about 80% to 90% of the total encoding time.

Table 2.1

Execution Time profiling of JMVC: Contribution of full search block matching in ME and DE for video sequence“Ballroom”

View ID 0 1 2

Search Range ±32 ±64 ±128 ±32 ±64 ±128 ±32 ±64 ±128 DE + ME % Time 97.25 98.73 99.76 99.25 99.48 99.84 97.74 99.38 99.80

When combining ME and DE over multiple views, block matching takes even longer, consuming almost the entire computation time. Table 2.1 shows the profiling of MVC test-bench, the joint multiview video coding (JMVC) software suite [27] for three

views and for three search ranges, taking up to 99.80% of the computation time. To accelerate ME/DE in the block matching, many sub-optimal techniques have been proposed in literature [8], [9]. These algorithms trade video quality and bit-rate for faster computation time. JMVC implements four sub-optimal search algorithms. They are full search, log search, spiral search and TZsearch. Among the sub-optimal algorithms for MVC, TZsearch provides the fastest implementation with the lowest degradation in terms of peak signal to noise ratio (PSNR) and bit-rate. Because of its superiority, TZsearch is also adopted for the next generation video coding standard (H.265) test software, high efficiency video coding (HEVC) test model (HM) [2],[28]. While a highly efficient algorithm, such as TZsearch, is sufficient for model testing, it does not provide a solution for real-time implementation of complex coding techniques needed for MVC.

To ease the coding complexity, application specific integrated circuit (ASIC) solutions are proposed for MVC [29], [30]. However, such hardware designs require significant simplification of coding process. The work in [29], as an example, presents the design

of a MVC on an ASIC platform. To achieve a high processing rate, the design

has made several major simplifications to the MVC coding when compared with the MVC reference software, JMVC, resulting in significant degradation in bit-rate and PSNR, defeating the purpose of achieving high coding efficiency in MVC. The adoption of simple prediction structure in the temporal domain, and small search region for the fast ME/DE algorithm [31], [32], [33] results in about 2.5 to 3 dB degradation in PSNR over the full range of bit-rates. In addition, the architecture in [29], to implement an efficient pipeline for processing video sequences from multiple views, uses the original image pixels instead of reconstructed pixels for prediction,

only a small fraction of reference frames (15%). For the other 85% of reference frames, TZsearch over the full search region is replaced by a block matching using a maximum of 15 predictors from the neighborhood in spatial (within the current frame), temporal (same video sequence) and inter-view (neighboring video sequences) domains. This simplification leads to an 11% increase in bit-rate and 0.11 dB in PSNR loss [30]. In general, ASIC implementations achieve their high frame rates due to their highly optimized architecture, at the expense of reduced flexibility, programmability and scalability.

The availability of cost-effective MPA computing platforms [34], [35], provides an op- portunity for the development of MVC parallel algorithms that are fast, and nearly- optimum, without sacrificing video quality or bit-rate. Characteristics of MPA platforms are the availability of a large number of computing cores that are generally organized as single instruction multiple data (SIMD), or single program multiple data (SPMD) computing paradigm. MPA embeds several blocks of fast shared memory allowing efficient coordination of multiple streams of the same program by the computing cores. In addition, MPA gains unprecedented performance through high bandwidth memory (Gigabytes per second (GB/s)).

There are exploratory implementations of block matching algorithms on the MPA platform of graphical processing units (GPU) [17], [13], [15]. All these implementations, except [15], are proposed for single view video, and have not even been integrated into the complex platform of JMVC, and therefore, not compared with TZsearch, (which is the gold standard for MVC). Even reference [15], the only paper on the GPU implementation of MVC, does not report the prediction model (see next section). Further, this paper reports the comparison with the outdated EPZS [11] which is no longer a search option in the JMVC software suite [27]. These algorithms, while demonstrating the potential of MPA, do not even exhibit impressive performance compared to the CPU-based full search. The best performance speedup

with respect to sequential full search reported in these references range from 9 to 17. As will be shown in this chapter, to gain an advantage over TZsearch an improvement in speedup over the sequential full search by at least one order of magnitude over the existing reported implementations is needed.

The main reason for the poor performance of algorithms in [17], [13], [15] is that they lack due consideration to the algorithmic scalability within the capabilities of the underlying hardware, and the inefficient use of memory bandwidth, resulting in unnecessary resource saturation and scaling limitations. These works achieve large scale parallelism by processing huge number of macroblocks simultaneously, thereby, saturating the resources of an MPA. However, they fail to process each macroblock in an efficient manner, by performing many parallelizable tasks sequentially. Further, these algorithms gain speedup at the expense of bit-rate and PSNR. This is due to the fact that, due to the hardware limitation of MPA, the parallel processing of a large number of macroblocks comes at the expense of significant reduction of the size of the search region (e.g. 20.48 fold reduction [15]). Further, parallel processing of macroblocks, as employed in [17], [13], [15], results in poor handling of spatial MV/DV dependencies in MVC, as required by H.264/AVC. (See Sec. 2.3). From our investigation the best implementation of MVC, to-date, in terms of speed, bit- rate, and PSNR quality, is the JMVC reference software with TZsearch mode for ME/DE, on a single-thread, single-core CPU. Therefore, all performance evaluations of proposed algorithms are made with respect to JMVC reference software as an anchor.

improvement with respect to the existing sequential full search and best performing sub-optimum search algorithm (TZsearch) for MVC to reduce encoding time. This reduction in the execution time comes with no, or insignificant, degradation in pic- ture quality or bit-rate. Further, the scalability of the proposed algorithm to multiple MPA computing units when they are available, is demonstrated.

In this work it will be shown that parallel processing, even on an MPA with hundreds of cores, is not just a na¨ıve parallel implementation of an algorithm such as parallel full search. The development of efficient fast parallel algorithms, similar to their sequential counterparts, requires a careful analysis of the multimedia algorithm, with a good understanding of the resource features and limitations of the underlying MPA.

This chapter is organized as follows. Section 2.2 presents a brief description of MVC prediction structures. Section 2.3 discusses the opportunities for massive parallelism in the full search algorithm. It also presents the implementation and performance results of the massively parallel full search. Further, it highlights the limitations of na¨ıve parallel full parallel search implementation. Section 2.4 presents the proposal for the massively parallel fast search algorithm. Section 2.5 covers the performance analysis for the proposed algorithm, and compares the results with alternative algorithms. Section 2.6 discusses the applicability of the proposed technique to HEVC.

!"# !$# !%# !&# !'# !(# !)# !*# !+# !,# !$"# !$$# -"# -$# -*# -%# -&# -'# -(# -)#

."# /&# /%# /&# /$# /&# /%# /&# ."# /&# /%# /&#

."# / /%# /&# /$# /%# ."# /&# /%# /&#

0"# /%# /& /$# /& /%# /& 0"# /& /%# /&# "# /&# /%# /&# $# /&# /%# /&# "# /&# /%# /&#

/&# /%# /&# $# /&# %# /&# # /&# /%# /&#

."# /&# /%# /&# /$# /&# /%# /&# ."# /&# /%# /&#

!"# !$# !%# !&# !'# !(# !)# !*# !+# !,# !$"# !$$# -"# -$# -*# -%# -&# -'# -(# -)#

0"# /&# /%# /&# /$# /&# /%# /&# 0"# /&# /%# /&#

0"# &# /%# / /$# / /%# /&# 0"# /&# /%# /&#

0"# /&# /%# /& /$# /& /%# /& 0"# /& /%# /&#

0"# /&# /%# /&# $# /&# /%# /&# 0"# /&# %# /&#

0"# /&# %# /&# $# /&# %# /&# 0"# /&# /%# /&#

0"# /&# /%# /&# /$# /&# /%# /&# 0"# /&# /%# /&#

!"# !$# !%# !&# !'# !(# !)# !*# !+# !,# !$"# !$$# -"# -$# -*# -%# -&# -'# -(# -)#

0"# &# /%# / /$# / /%# /&# 0"# /&# /%# /&#

/$# / /&# /' /%# / /&# /$# /&# /'#

."# /&# %# /&# # /&# /&# "# /&# %# /&#

/ /'# /&# /'# /'# /&# /'# /$# /'# /&# /'#

."# /&# /%# /&# /$# /&# /%# /&# ."# /&# /%# /&#

/$# /'# /&# /'# /%# /'# /&# /'# /$# /'# /&# /'#

."# /&# /%# /&# /$# /&# /%# /&# ."# /&# /%# /&#

!"# !$# !%# !&# !'# !(# !)# !*# !+# !,# !$"# !$$# -"# -$# -*# -%# -&# -'# -(# -)#

0"# / /%# /&# /$# /&# /%# 0"# /&# /%# /&#

."# / /%# / /$# / /%# / ."# /%# /&#

/&# /%# /&# # /&# /%# /&# "# /&# /%# /&# "# /&# /%# /&# /$# /&# /%# /&# "# /&# /%# /&#

."# /&# /%# /&# /$# /&# /%# /&# ."# /&# /%# /&#

123#-4567829:# 1;3#.0.#

183#0/.# 1<3#0..#

Figure 2.1: Multiview prediction structures: (a) Simulcast: independently coded without inter-view prediction, (b) Inter-view prediction for key frames only with PIP structure, (c) Inter-view prediction for all key and non-key frames with IBP structure, (d) Inter-view prediction for all key and non-key frames with IPP structure.

In document High Performance Multiview Video Coding (Page 54-62)