for Image Compression
with Application to Medical Images
Ph.D. dissertation
Alexandre Krivoulets
IT University of Copenhagen
February 18, 2004
This thesis covers different topics on design of image compression algorithms. The main focus in this work is development of efficient entropy coding algorithms, development of optimization techniques for context modeling (with respect to the minimum code length) and application of the methods in the design of an algorithm for compression of medical images.
Specifically, we study entropy coding methods based on a technique of binary de- composition of source symbols. We show that the binarization allows to fit a parametric probability distribution model to a source coding algorithm, e.g., arithmetic coding, reducing the number of coding parameters to that of the distribution model.
Context modeling is an essential part of an image coding algorithm, which basically defines its compression performance. In the thesis, we describe a unified approach to this problem based on statistics learning methods and the minimum description length principle. In particular, we present a design of optimized models using context quantization and initialization techniques. The optimization allows to find a model, which yields the minimum code length for some set of training data samples.
Entropy coding and context modeling methods are applied for developing a com- pression algorithm intended for medical images. The algorithm allows for progressive near-lossless coding and is based on lossy plus refinement layered approach. We show that this method results in a better compression performance and image quality for large distortion values compared with the recently adopted standard JPEG-LS for lossless and near-lossless image compression. We also investigate a possibility of image reconstruction with the minimum mean squared error criterion within the proposed framework.
i
Contents iii
1 Introduction 1
1.1 Motivation and main goals . . . 1
1.2 Previous work . . . 2
1.3 About this thesis . . . 3
2 Source coding and image compression 7 2.1 Information sources . . . 7
2.2 Source coding . . . 8
2.3 Arithmetic coding . . . 10
2.4 Universal source coding . . . 11
2.5 Compression of images . . . 13
3 Source coding via binary decomposition 17 3.1 Introduction . . . 17
3.2 The binary decomposition technique . . . 18
3.3 On redundancy of binary decomposition . . . 19
3.4 Binary decomposition of FSM sources . . . 21
3.5 Binary decomposition and universal coding . . . 21
4 Applications of binary decomposition 25 4.1 Introduction . . . 25
4.2 Generalized two-sided geometric distribution . . . 25
4.3 Efficient coding of sources with the GTSGD . . . 27
4.4 Experimental results . . . 32
4.5 On redundancy of rice coding . . . 35
4.6 Summary . . . 38
5 Context modeling for image compression 41 5.1 Introduction . . . 41
5.2 Context formation . . . 43
5.3 Context model optimization . . . 44
5.4 Context initialization . . . 46
5.5 Context quantization . . . 49
5.6 Summary . . . 53 iii
6 Optimization in the JPEG2000 standard 55
6.1 Introduction . . . 55
6.2 Context modeling in JPEG2000 . . . 56
6.3 High-order context modeling . . . 58
6.4 Experimental results . . . 61
6.5 Summary . . . 62
7 Hierarchical modeling 65 7.1 Introduction . . . 65
7.2 Two-stage quantization . . . 66
7.3 Tree-structured quantization . . . 67
7.4 Optimal tree pruning . . . 67
7.5 Experimental results . . . 68
7.6 Summary . . . 69
8 Compression of medical images 71 8.1 Introduction . . . 71
8.2 Embedded near-lossless quantization . . . 74
8.3 Entropy coding of the refinement layers . . . 76
8.4 Experimental results . . . 80
8.5 Reconstruction with minimum MSE criterion . . . 82
8.6 Summary . . . 86
Bibliography 89
A Bit rates for test sets of medical images 95
B PSNR for test sets of medical images 117
Introduction
1.1 Motivation and main goals
Computer imaging plays a significant role in many areas ranging from consumer dig- ital photo albums to remote earth sensing. The growing production of images and demands to their quality require high performance compression methods for efficient transmission, storage and archival.
Most image compression algorithms can be viewed as consisting of an image trans- formation, which performs decomposition of an image into a sequence of descriptors, followed by entropy coding of the descriptors. Entropy coding, in essence, performs the compression.
Prediction, discrete cosine and wavelet transforms are examples of image decom- position, whereas prediction errors and transform coefficients are examples of the de- scriptors. They constitute an information source for the entropy coder, which encodes the sequence of source symbols according to some source model. This model is nor- mally designed off-line using some assumptions on the data to be coded. The better the model approximates statistical properties of the data, the higher compression can be achieved. The lower bound on the compression performance is defined by the en- tropy of the source. The design of efficient models is a task of primary interest for any compression algorithm.
Most image compression algorithms use the universal coding approach, where the model has fixed structure and unknown parameters. The parameters are estimated on the fly during encoding. This approach allows to adopt the model to the data statistics, which may vary for different data. On the other hand, the price for that adaptivity is the increase of the code length due to the necessity of implicit or explicit transmission of information on the model parameters (the so called model cost). The higher the model order, the more parameters it involves, and the higher the model cost. Thus, there is a trade-off between the order of the model and the overhead data needed to specify the model parameters. The use of domain knowledge of the data allows for reduction of the overhead information. Finding optimal solutions to this problem is one of the goals of this thesis.
In this connection, we investigate the technique of source coding via binary de- composition of symbols. The decomposition can be used as a mean for model cost
1
reduction and/or for efficient model optimization.
Another goal is the design of an efficient algorithm for compression of medical images. The design of such an algorithm differs from the design of general purpose image compression methods due to some specific requirements.
The main concern in medical imaging is the quality of the reconstructed image.
There are three kinds of image compression methods: lossless, lossy and near-lossless.
Lossless compression methods perfectly reconstruct an image, but they compress data at the lowest degree.
Lossy compression techniques are usually used for images intended for human ob- servation in general-purpose computer systems. Such methods are based on the fact that the human visual system does not perceive the high frequency spatial components of the image, and those components may be removed without any visible degrada- tion. Thus, introducing distortion in the reconstructed data, one can achieve higher compression, while preserving its visual information content. Lossy methods have the main advantage that they can compress images at the highest degree. However, the distortion of the data is undesirable in some applications, e.g., in image analysis, object recognition and some others.
Near-lossless compression is a lossy image coding algorithm, where the distortion is specified by the maximum allowable absolute difference (the tolerance value) between pixel values of the original and the reconstructed images. The method allows for a rigid and robust control of errors while achieving reasonable compression performance.
For that reason near-lossless compression seems to be an attractive method for medical images. In our work, we develop an efficient algorithm, which allows for pro- gressive near-lossless compression up to the lossless mode. The extended functionality and makes the algorithm more suitable for real applications.
Compression of medical images also allows for efficient use of the similarity of statistical properties of the data to design high-order source models and thus, to achieve better compression performance.
1.2 Previous work
Image coding originates from the communication theory established in 1948 by Shannon in his famous paper [50]. Yet, it took years before the methods developed in this work could be used in practical algorithms for image compression.
In the recent decades, image compression has been extensively studied by many researchers. The first lossy and lossless still image coding standard JPEG [33] appeared by the end of 1980’s and had a remarkable performance at that time. The standard uses a block discrete cosine transform (DCT). The DCT was shown to be very close to the optimal decomposition for the class of natural images, performing almost perfect decorrelation of image pixels, see, e.g., [35]. The standard uses an ad-hoc Huffman or arithmetic entropy coding of transform coefficients. The version with arithmetic coding exploits binarization of the coefficients and a simple heuristic context modeling. The JPEG standard remains the most popular image compression algorithm.
Introduction of a (discrete) wavelet transform (DWT) in the middle of 1980’s [20,
30, 26] launched a new era in image compression. Besides being good for decorrelation of image pixels, the new transform provides more functionality to the compression algorithm, such as embedded coding, where any initial part of the compressed bit stream can be used to reconstruct the image with the quality in proportion to the length of this part. A benchmark method called the embedded zerotree wavelet (EZW) coding was introduced in [51] and further developed in the SPIHT (Set Partitioning in Hierarchical Trees) algorithm [47]. The use of integer-to-integer (reversible) wavelet transform [9] allowed for progressive coding up to the lossless image reconstruction [8, 48, 3, 64].
Development of context modeling techniques and rate-distortion optimization lead to a remarkable improvement of compression efficiency of algorithms based on DWT.
The most sophisticated methods are the high-order embedded entropy coder of wavelet coefficients (ECECOW) [64], embedded block coding with optimized trancation (EBCOT) [55], and the compression with reversible embedded wavelets (CREW) al- gorithm [69], just to name a few. The algorithm EBCOT was adopted as a basis for the new standard for still image coding JPEG2000 [56].
The best compression results for lossless image compression were obtained by algo- rithms based on predictive coding and context modeling of prediction residuals. For example, algorithm CALIC (Context-based, Adaptive, Lossless Image Codec) [68] uses a sophisticated context modeling and a special kind of prediction (it exploits non-linear, gradient adaptive predictor) to achieve the best compression performance among prac- tical coders. The performance of CALIC turns out to be even better than the UCM (Universal Context Modeling) method [57] developed on the basis of a universal source coding algorithm with provable asymptotic optimality.
The ideas from CALIC were employed in the LOCO-I (LOw COmplexity LOssless COmpression for Images) algorithm [59], which became the new standard for lossless image coding JPEG-LS [60]. The standard also supports near-lossless compression.
Near-lossless compression first appeared in [10] as a method allowing for much better compression performance compared to the lossless coding at the price of a very small and controllable distortion of the image. It was further elaborated in [65] to achieve higher compression efficiency. The fist method allowing progressive coding was proposed in [5].
The major improvements in compression performance are due to development of new techniques for context modeling, such as context quantization [67], and the use of parametrized probability distribution models (LOCO-I).
1.3 About this thesis
The thesis covers two major problems in image compression: entropy coding and con- text modeling. For efficient entropy coding we studied the use of a binary decompo- sition technique with application to coding of sources with parametrized distribution.
For design of high-order context models we developed optimization methods based on training data samples.
The proposed solutions are tested in a series of experiments and applied in the design
of an algorithm for near-lossless compression primarily intended for coding medical images.
The thesis is organized such that most chapters are self-contained is the sense that they cover different topics, yet in the same framework. The material of the thesis is based on the following papers:
1. A. Krivoulets, “On redundancy of coding using binary tree decomposition”, in Proc. 2002 IEEE Int. Workshop on Inf. Theory, p. 200, 20-25 Oct., Bangalore, India, 2002. (Chapter 3)
2. V.F. Babkin, A.G. Krivoulets, “On coding of sources with Laplacian distribu- tion,” in Proc. of Popov’s Society Conference, p. 254, May, 2000, Moscow, Russia.
(in Russian) (Chapter 4)
3. A. Krivoulets, “On coding of sources with two sided geometric distribution using binary decomposition”, in Proc. 2002 Data Compression Conf., p. 459, Snowbird, UT, Apr., 2002. (Chapter 4)
4. A. Krivoulets, “Efficient entropy coding for image compression,” IT University of Copenhagen, Tech. report TR-2002-13, February, 2002. (Chapter 4)
5. A. Krivoulets, “Fast and efficient coding of low entropy sources with two-sided geometric distribution,” in Proc. of the 2nd European workshop on Advanced video-based surveillance systems, pp. 183-188, Kingston, UK, Sep., 2001. (Chap- ter 4)
6. A. Krivoulets “On redundancy of Rice coding,” IT University of Copenhagen, Tech. report TR-2003-32, September, 2003. (Chapter 4)
7. A. Krivoulets, X. Wu, and S. Forchhammer, “On optimality of context modeling for bit-plane entropy coding in the JPEG2000 standard”, In Proc. VLBV03 Workshop, pp. 208-216 Madrid, Spain, Sep., 2003, (LNCS 2849) (Chapters 5,6) 8. A. Krivoulets and X. Wu, “Hierarchical modeling via optimal context quantiza-
tion”, in Proc. 12th International Conference on Image Analysis and Processing, pp. 380-384, Mantova, Italy, Sep., 2003. (Chapter 7)
9. A. Krivoulets, “A method for progressive near-lossless image compression”, in Proc. ICIP2003, vol. 2, pp. 185-188, Barcelona, Spain, Sep., 2003. (Chapter 8) 10. A. Krivoulets, “Progressive near-lossless coding of medical images”, in Proc. 3d
International Symposium on Image and Signal Processing and Analysis, vol. 1, pp. 202-207, Rome, Italy, Sep., 2003. (Chapter 8)
A brief description of the chapters is as follows. Chapter 2 describes the main con- cepts of image compression algorithms and introduces the framework of our research.
In this chapter, we describe basic principles and the place of entropy coding and context modeling in image compression algorithms.
In Chapter 3, we discuss a technique of binary decomposition of source symbols as an efficient mean for entropy coding and present some properties of the technique.
Some applications of the technique are presented and discussed in Chapter 4. In this chapter, we introduce a probabilistic model of sources, which often occurs in image compression, and describe efficient methods of coding of such sources using the binarization technique. Binarization allows to reduce the number of coding parameters, which is usually equal to the size of the source alphabet, to the number of probability distribution parameters, which normally is much lower. Binarization also simplifies context model optimization described in the next chapter.
Chapter 5 presents basic principles of high-order context modeling for image com- pression. The chapter covers two main topics of the context model design: context formation and context model optimization. The latter is based on context initializa- tion and quantization based on some prior statistics on the data.
In Chapter 6, we describe an application of the high-order context modeling tech- niques developed in Chapter 5 to optimization of context models adopted in the JPEG2000 standard. The models are used in the bit-plane entropy coding of wavelet transform coefficients and basically define the compression performance of the stan- dard. In this chapter we demonstrate “almost optimality” of the models adopted in JPEG2000 with the given context template.
Chapter 7 extends ideas of the context quantization technique to build a hierarchical set of models intended for a better fit to the actual data.
Finally, in Chapter 8, we present an algorithm for near-lossless compression intended for medical images, which allows for progressive coding and reconstruction. In the algorithm design, we exploited the methods developed in Chapters 3,4, and 5. We show, that the resulting algorithm allows for more efficient coding (in terms of both compression performance and functionality) than the recently adopted standard JPEG- LS for lossless and near-lossless image compression.
Source coding and image compression
In this chapter we introduce the basic theory, concepts and definitions of source coding and consider a general structure of image compression algorithms. This background will be used throughout the thesis.
2.1 Information sources
By an information source we assume a mechanism generating discrete random variables u from a countable (often finite, but maybe infinite) set A = {a0, a1, . . . , am−1}. The set A is called the source alphabet and m = |A| defines the alphabet size (hereafter,
| · | denotes cardinality of a set or the length of a string).
Let ut be a random variable generated at the time instance t. A string of source symbols of length n (a source message) is denoted as un1 = u1u2u3. . . un. The empty string u01 is denoted as ∅. Let An be the set of all possible messages of length n in the alphabet A: An={un1}.
The simplest source model generates an independent and identically distributed (i.i.d.) symbols according to the probability distribution P (A) ={p(a), a ∈ A}, which defines the model parameters (probabilities of source symbols). Such a model is called the memoryless source. The distribution P (A) does not depend on the past symbols.
A general source model is a finite state machine (FSM) model, which is defined by the (finite) set of states S = {s}, the source alphabet A, the set of conditional probability distributions {Ps(A), s∈ S}, Ps(A) ={p(a|s), a ∈ A}, and the initial state s1. At each instance of time t the source generates a symbol ut ∈ A and changes its state from st to st+1 according to the state transition rule
st+1 = F (st, ut1). (2.1)
The function F (·) will be referred to as a model structure. The set of probability distributions {Ps(A), s∈ S} specifies the set of model parameters.
A message un1 generated by an FSM source can be decomposed into|S| subsequences generated at the states s ∈ S, which are the i.i.d. sequences drawn according to the distribution Ps(A).
7
If the state st+1 is uniquely defined by the previous state st and the symbol ut
generated at this state, i.e.,
st+1 = F (st, ut), (2.2)
then the model is called a Markov source. The structure function F (·) of a Markov source can be described by a directed graph of state transitions, where nodes define the states and edges correspond to the source symbols and define state transitions.
Let Pij be the probability of entering the state s = j from the state s = i:
Pij = Pr(st+1 = j|st = i). (2.3) If the state transition rule F (·) is such that the probability (2.3) depends only on the previous state, i.e.,
Pr(st+1|st, st−1, . . . ) = Pr(st+1|st),
then the sequence of the states form a homogeneous Markov chain. If the state is uniquely defined by the o last source symbols utt−o+1, then the source is called an o-order Markov chain. For the o-order Markov chain model we have:
p(a|s) = p(a|utt−o+1) and
st+1 = F (utt−o+1), where
F (utt−o+1) : utt−o+1 7→ s ∈ {1, 2, . . . , |Ao|}.
The FSM model is a good approximation to most practical sources, therefore it is widely used in compression algorithms.
The main property of an information source is its entropy defined as H = lim
n→∞
1
nH(An), (2.4)
where
H(An) =− X
p(un1)∈An
p(un1) log2p(un1), (2.5)
and p(un1) is the probability of the string un1.
2.2 Source coding
Let B = {0, 1} be a binary alphabet and B∗ be a set of all words in the alphabet B. Source coding is a mapping, which to each source message un1 ∈ An assigns a codeword ϕ(un1)⊂ B∗ such that un1 can be uniquely reconstructed. A set of codewords {ϕ(un1), un1 ∈ An} is called a prefix code if no one codeword is a prefix of any other in the set. The use of prefix code guarantees the unique decipherability.
The codeword lengths |ϕ(·)| of any set of prefix codes must satisfy the Kraft in- equality:
X
un1∈An
2−|ϕ(un1)| ≤ 1. (2.6)
Conversely, if the set of codeword lengths satisfies (2.6), then there exists a prefix code with these codeword lengths.
The set Q(An) = {q(un1) = 2−|ϕ(un1)|, un1 ∈ An} is called the coding probability distribution on An. Since the codeword lengths are of main concern, this distribution is often useful in the source coding analysis.
Given the source model, the key challenges in source coding are the choice of the codeword lengths and the construction of the codewords.
Normally the codewords are chosen to minimize the description length |ϕ(un1)|, meaning data compression. Thus, the terms source coding and data compression are often interchanged in the literature, even though source coding has a broader meaning.
In the future, we will follow the tradition and use both terms in the same sense.
Let
L = X
un1∈An
p(un1)|ϕ(un1)| (2.7) be the average code length for the set An. The source coding theorem ([16], Theo- rem 3.3.1) establishes the lower bound on L for lossless coding
L ≥ H(An), (2.8)
with equality iff |ϕ(un1)| = − log2p(un1). The theorem also states that there exists a prefix code such thatL < H(An) + 1. Thus, the minimum possible code length for the message un1 is defined by
|ϕ(un1)| = − log2p(un1). (2.9) This quantity is called the self-information of the message un1.
This is the basic idea of source coding: data compression is possible by assign- ing shorter codewords to more probable messages (symbols) and longer codewords to less probable messages (symbols). Maximum compression is achieved by choosing the codeword lengths equal to minus the logarithm of the probability of a message.
The main property of a code is its redundancy. There are different redundancy measures defined in source coding. The two measures, which will be used later in the thesis, are the average redundancy
Ra=L − H(An), (2.10)
and the individual redundancy
Ri =|ϕ(un1)| + log2p(un1), (2.11) respectively.
The task of a codeword construction can be solved by using arithmetic coding.
2.3 Arithmetic coding
Arithmetic coding is a method for sequential calculation of the codeword ϕ(un1) for the source message un1. It is based on unpublished Elias’ algorithm described by Abramson [2] and Jelinec [22]. First practical implementations are due to Rissanen [43] and Pasco [32], who solved the finite precision problem and Witten et al. [61] who made it popular by publishing the C-code of their implementation.
In arithmetic coding, the codeword is recursively calculated as a cumulative coding probability of the string un1:
ϕ(un1) = X
a<u1
q(a|∅) + q(u1)X
a<u2
q(a|u1) +· · · + q(un−11 ) X
a<un
q(a|un−11 ), (2.12)
where {qt(a|ut1), a ∈ A, t = 0, 1, . . . , n − 1} is a sequence of conditional coding proba- bility distributions satisfying
qt(a|ut1) = q(ut1a)
q(ut1) , (2.13)
X
a∈A
qt(a|ut1)≤ 1, (2.14)
such that
q(un1) =
n−1Y
t=0
q(ut+1|ut1). (2.15)
The calculations are assumed to have infinite precision resulting in ideal arithmetic coding. The main property of ideal arithmetic coding is the following theorem, which is a slightly modified version of Theorem 1 from [52].
Theorem 1. Given a sequence of coding distributions {qt(a|ut1), a∈ A, t = 1, 2, . . . , n}
satisfying (2.13), (2.14), an arithmetic coder achieves codeword lengths
|ϕ(un1)| < − log2q(u1n) + 2, ∀un1 ∈ An. (2.16) The codewords form a prefix code.
In practice, calculations are performed with finite precision. In this case, estimates on upper bounds of the coding redundancy are defined in the following theorem.
Theorem 2. Let r bits be used in a binary representation of the coding probabilities q(a|·), a ∈ A, and g ≥ r + 2-bit registers for calculations. Then
Ra≤ mn(r + log2e)2−(g−2)+ 2, (2.17)
Ri ≤ g + (n − 1)2r−g. (2.18)
q(ut+1|ut1) q(ut+1|ut1) ϕ(u)
u u
'
&
$
%
'
&
$
%
ENCODER DECODER
- - -
-
6 6
Arithmetic coder
Source model
Arithmetic coder
Source model
Figure 2.1: A block-chart of source coding using arithmetic coder.
This theorem essentially is a compilation of Theorem 4 form [46], which establishes the bound for Ra, and the estimate on Ri given in [41]. It is clear from the theorem, that even with finite precision arithmetic implementation, the coding redundancy is usually negligible.
Using arithmetic coding, we can separate the problem of assigning the coding prob- abilities q(ut+1|ut1), according to some chosen criterion (different criteria will be con- sidered in the next section), from the codeword construction.
Source coding using arithmetic coder is schematically represented on Figure 2.1 [1], where the source model does the job of sequential assigning the coding probabilities q(ut+1|ut1) and the arithmetic coder performs sequential calculation of the correspond- ing code word.
2.4 Universal source coding
If the source model is known, then the choice of the coding distribution q(a|·) = p(a|·), a ∈ A allows to calculate a codeword for the message un1 using arithmetic coding exceeding the ideal by at most 2 bits (disregarding the arithmetic precision problem).
However in practice, the source parameters (or even the underlying source model) are usually not known in advance or may vary for different data.
In universal coding, it is assumed that the source belongs to some predefined set of models Ω = {ω}. A code is designed to perform well for all, or most of the models in the set. The set can be just a parametric probabilistic set of sources having the same model structure (e.g., Markov source with the same state transition rule) or may be a double mixture of sources with different structure and a set of parameters [11].
Let ϕΩ(An) ∈ B∗ be a prefix code on An used for all sources in the set Ω and R{ϕΩ(An)} be a measure of performance (redundancy) of the code such that
R{ϕΩ(An)} ≥ 0, (2.19)
with equality when the set Ω contains only a single element, i.e., when the source is
known (the source structure and the parameters are given). Thus, the redundancy of universal coding is only due to the lack of knowledge about the source.
Let
˜
ϕΩ = arg inf
ϕ R{ϕΩ(An)} (2.20)
and
R∗ =R{ ˜ϕΩ(An)}. (2.21)
The code ˜ϕΩ is said to be universal w.r.t the model set Ω if
R∗ → 0 (2.22)
as n→ ∞ [11].
There are two main measures used in universal coding, which are based on the average and individual redundancy. Let
Rna(ϕΩ, ω) = X
p(un1|ω)∈An
p(un1|ω)|ϕΩ(un1)| − H(An|ω) (2.23)
and
Rni(ϕΩ, ω) =|ϕΩ(un1)| + log2p(un1|ω) (2.24) be the average and individual redundancy of the code ϕΩ for the source ω, respectively.
Then the measures of universal coding are defined by the maximum average Ra{ϕΩ(An)} = sup
ω∈Ω
1
nRan(ϕΩ, ω) (2.25)
and individual
Ri{ϕΩ(An)} = sup
ω∈Ω
1
nRni(ϕΩ, ω) (2.26)
per-symbol redundancy.
In general, the use of different criteria results in different codes. The criterion (2.26) was first proposed in [53, 54]. Clearly, it is stronger than (2.25) and a code ˜ϕΩ with
“good” properties according to the criterion (2.26) also implies good behavior w.r.t.
the criterion (2.25).
The lower bound on the convergence (2.22) w.r.t. to both criteria for the parametric sets of memoryless and FSM sources is defined by [44, 52]
R∗ ≥ K
2nlog2n + O 1 n
, (2.27)
where K is the number of free parameters. For memoryless sources K = m− 1, and for FSM sources with |S| states K = |S|(m − 1) (m is the alphabet size).
Coding of an FSM source using the sequential coding probability distribution qt+1(a|s) = ϑ(a|ut1s(s)) + αa(s)
ts+P
a∈Aαa(s) , (2.28)
combined with arithmetic coding, where ϑ(a|ut1s(s)) is the number of occurrences of the symbol a in the subsequence ut1s(s) corresponding the state s, and ts is the length of the subsequence, allows to achieve the optimal convergence rate (2.27).
The values {αa(s) > 0, a ∈ A, s ∈ S} define prior distributions on the source parameters {Ps(A), s∈ S}. If nothing is known about P (A)s, then the best choice is αa(s) = 12, a ∈ A, s ∈ S, see, e.g., [23, 54].
For memoryless sources, (2.28) is reduced to qt+1(a) = ϑ(a|ut1) + αa
t +P
a∈Aαa
. (2.29)
2.5 Compression of images
A gray-scale digital image1 is a 2-dimensional array of bounded integer values (image pixels) v[y, x], 1 ≤ x ≤ X < ∞, 1 ≤ y ≤ Y < ∞, where y and x define the row and the column coordinates, respectively. The values X and Y define the size of an image. The range of pixel’s values is normally defined in terms of the number of bits k required to represent all image pixels. Thus, for a k-bit image, the pixel’s value lies in the range2 v[y, x] ∈ [0, 2k− 1]. Most general purpose images use 8-bit representation.
Medical images often use 10 . . . 16-bit representation. An example of an image is shown on Figure 2.2, where the small squares represent pixels and the grey scale reflects the pixel’s value.
In its original representation, an image takes X× Y × k bits. Image compression aims at reducing this number by using source coding techniques.
An image compression system can be described in terms of four functional blocks:
image transformation, quantization, encoder model (not to be confused with the source model introduced in Section 2.2), and entropy coder, as it is shown on Figure 2.3, where the mandatory part is the entropy coder. (In general, the image pixels can be fed directly to the entropy coder, however, in most applications this approach would lead to inferior compression performance due to significant spatial correlation of the image pixels.)
The image transform part converts an image into a sequence of descriptors being another (abstract) representation of the image. The most used transformation tech- niques are prediction and a discrete orthogonal transform. The transformations use the fact that image pixels typically have substantial correlation with their neighbors.
Predictive coding exploits an auto-regressive model, where the weighted past image pixels is used to predict the value of the next pixel. The difference values (residuals) between the estimated (predicted) and the original pixel values define the sequence of descriptors.
The idea of an orthogonal transformation is that the image is represented as a linear combination of some basis functions (waveforms). The transform coefficients are the weighting factors to these functions. If the transform coefficients are real values, then quantization of the coefficients is a necessary step of image coding.
1In the thesis, we will deal only with 2-D images.
2Without loss of generality, we assume that pixels can take only non-negative values.
50 100 150 200 250
5 10 15 20 25 30 35 40 45 50
5
10
15
20
25
30
35
40
45
? 50
-
y
x
v[y, x]
Figure 2.2: Example of a gray-scale 8-bit image of size 50× 50 pixels.
The zero-order entropy of prediction residuals and (quantized) transform coefficients is much lower than that of image pixels. Furthermore, using orthogonal transformation, an image can be represented by a few transform coefficients. All that significantly reduces the amount of storing or transmitting data.
The most used kinds of orthogonal transformation for compression are the discrete cosine and wavelet transforms (DCT and DWT, respectively) [36]. Basis functions of the DCT are asymptotically optimal for decorrelation of the 1-st order 1-dimensional stationary Markov process. Nevertheless, 2-D separable transformation still has a good decorrelation property and allows for a high compression capability (retaining most information in a few transform coefficients). There exist algorithms for fast calculation.
The DCT is used in many image and video compression algorithms. Examples are the still image coding standard JPEG (Joint Picture Expert Group) [33], and the video coding standard MPEG (Motion Picture Expert Group) [25, 49].
The wavelet transform has a superior energy compaction due to comparatively short basis functions. It fits better to a non-stationary signal like real images. Yet, the main advantage of using DWT is its inherited ability for multiresolution representation of the signal, which adds an additional useful functionality to a compression algorithm.
One more benefit of the DWT is that there are reversible integer-to-integer trans- formation algorithms [9]. Using embedded quantization of the transform coefficients,
Original Image
Compressed Data
- Encoder -
' ?
&
$
%
Encoder
- Image - - - - - -
Transform Quantization Descriptors Encoder Model
Source
Symbols Entropy Coder
Figure 2.3: An image compression system.
such a transformation allows for an efficient progressive quality and spatial resolution representation up to perfect reconstruction of the original image.
The DWT is exploited in the new standard for compression of still images JPEG2000 [56]. Objective and subjective tests show that the DWT achieves higher compression performance then the DCT. However, the DWT is normally more time and memory consuming.
The transformation produces “almost” uncorrelated data and makes further quan- tization step more efficient, ultimately resulting in a better compression.
Quantization of the descriptors (prediction errors, transform coefficients) introduces distortion into the reconstructed image and results in lossy compression. Lossy com- pression allows for higher compression rates than the lossless coding. Using an appro- priate quantization rule, the quality of the reconstructed image and the compression rate can be efficiently managed and controlled. The theoretical principles of such a trade-off are given in the rate-distortion theory, see, e.g., [7, 16].
Quantization of the DCT or DWT transform coefficients yields an approximation of the original image in L2 sense. Quantization of prediction errors results in the so called near-lossless compression, where the distortion is specified by the maximum absolute difference between the original and the reconstructed image pixels. In the literature it is also called L∞ constrained lossy coding [65].
The encoder model maps the descriptors into source symbols for the entropy coder.
It may be done explicitly or implicitly. An example of explicit mapping is the (old) JPEG standard. In JPEG, the sequences of quantized high-frequency transform coeffi- cients are converted into blocks, which are then coded using Huffman code. Bit-plane coding of wavelet transform coefficients in the JPEG2000 standard is an example of implicit mapping, where the sequence of binary symbols is an information source for the entropy coder. In some coders the descriptors themselves (transform coefficients or prediction residuals) constitute input symbols for the entropy coder.
The transformation alone does not perform the data compression. On the contrary, it often performs an expansion of the data, since the transform coefficients, even quan- tized, may take more bits for their representation than the pixels themselves. The encoder model just translates the descriptors into a more convenient representation for the entropy coder and it does not change the amount of description information.
The compression is performed solely by the entropy coder. That is why developing an efficient entropy coder is an important step in the design of an image compression algorithm.
Source coding via binary decomposition
In this chapter, we study source coding using binary decomposition of source symbols and derive some properties.
3.1 Introduction
In Chapter 2 we introduced a general structure of image compression algorithms con- sisting of image transformation and entropy coding of description symbols (descriptors).
The sequence of descriptors constitutes an information source for the entropy coder.
In general, the definition of the information source in compression algorithms (the encoder model) is left to the algorithm designer. It can be a sequence of the descriptors itself. The source symbols can also be represented by blocks of the descriptors, like it is implemented in the JPEG standard for coding the AC coefficients of the DCT [33]. The grouping of symbols is called alphabet extension [42]. It is commonly used in combination with a Huffman coding. However, it was shown in [42] that any source can be coded without alphabet extension using an arithmetic coder.
An information source can also be reduced to a binary source via binary decom- position of source symbols. Binary decomposition combined with a binary arithmetic coding is a well known method for coding m-ary sources, see, e.g., [34, 24, 21]. The use of binary decomposition of (non-binary) source symbols has a number of advan- tages over conventional m-ary coding. Binary arithmetic coding is much simpler for hardware and software implementations than an m-ary arithmetic coding, especially if m 2. On the other hand, one has to encode more than one binary event for each in- put symbol. However, using an appropriate tree for decomposition and a binary coding technique, the method may result in a faster and/or more efficient entropy coding.
Another advantage of using binarization is the possibility of easy optimization of context models for conditional entropy coding, as will be described in Chapter 5.
In this chapter, we formally introduce the technique and present some properties.
17
3.2 The binary decomposition technique
Let us suppose a memoryless source with an alphabet A = {a0, a1, . . . , am−1}, and a probability distribution of source symbols P (A) = {p(a), a ∈ A}. Let B = {1, 0}
denote the binary alphabet and let An, Bn be sets of all words of length n in the alphabets A and B, respectively. As usually, a source message of length n is denoted as un1 = u1u2. . . un, u∈ A.
Let Tm be a set of proper and complete binary trees with m terminating nodes (leaves) χ0, χ1, . . . , χm−1 and Λ ={ηj, j = 0, 1, . . . , m− 2} be a set of m − 1 internal nodes of a tree τ ∈ Tm.
Let the tree τ be assigned to the source A in such a way, that to each symbol ak, k = 0, 1, . . . , m− 1, of the source, there corresponds a leaf χk of the tree τ :
τ : ak7→ χk. (3.1)
Then the source symbol ak is represented by a string of binary decisions, which is a path from the root to the leaf χk.
The m-ary (memoryless) source, generating a message un1 of length n in the alphabet A, can now be considered as a binary Markov source modeled by the tree τ generating a binary sequence b1b2. . . bn0 = b1n0 ∈ Bn0 of length n0, where b∈ B. Each node η ∈ Λ of the tree corresponds to a state of the Markov source and the tree uniquely defines a corresponding directed graph of state transitions. The initial state is defined by the root node. At each state (node), the source generates symbol 0 or 1 with the probability distribution p(0|η) and changes its state (maybe to the same state). The distributions {p(0|η), η ∈ Λ} are uniquely defined by the probability distribution of the alphabet P (A).
Such a binary source can be encoded by conventional methods of coding of Markov sources. Given the model and the initial state, the binary sequence bn10 is decomposed into m− 1 subsequences {bn10η(η), η∈ Λ}, generated at each state. The subsequences in essence are the sequences of i.i.d. binary symbols drawn with the probability p(0|η).
They can be effectively encoded using some kind of binary coding techniques, e.g., arithmetic coding [61] or Golomb run-length coding [19]. (Note, that for an arithmetic coder there is no need for an explicit decomposition of the binary sequence into the sub- sequences corresponding to the states. The arithmetic coder just uses the conditional probabilities p(0|η) to calculate a codeword for the whole binary sequence.)
The main parameter of a decomposition tree is the average number of binary coding operations per source symbol
n =X
a∈A
p(a)n(a), (3.2)
where n(a) is the number of binary decisions required to code a (the length of the binary path to the symbol a). This parameter defines the coding efficiency (in terms of both redundancy and speed, as will be discussed in the following sections). The minimum n is achieved when the tree is a Huffman tree for this source. The use of a
binary decomposition assumes that the decomposition tree is fixed1 during encoding, and compression is performed by a binary coding technique.
3.3 On redundancy of binary decomposition
In this section, we derive an upper bound estimate on the coding redundancy using the binarization technique.
We define a redundancy R as the difference between the average code length per source symbol and the entropy of the source:
R = 1 n
X
un1∈An
p(un1)|ϕ(un1)| − H, (3.3) where |ϕ(un1)| denotes the codeword length for the message un1 (see Chapter 2) and H =−P
a∈Ap(a) log p(a).
Let us allow the use of different binary coding techniques for each node. (For example, if some subsequence has probability of a binary symbol close to 0.5, then, for the sake of simplicity of implementation, this subsequence may be put directly into the output bit stream). If the probabilities of source symbols are unknown, then the subsequences can be coded using some kind of universal or adaptive coding technique (e.g., binary arithmetic coding with the symbol probability estimate given by (2.29).
Let n0η be the length of a binary sequence, generated at the state η. We define an average coding redundancy per binary symbol at the state η as
ρη = 1 n0η
X
bn0η1 ∈Bn0η
p(b1n0η)|ϕb(b1nη0)| − Hη, (3.4)
where|ϕb(bn10η)| is the code length of a binary string bn10η, generated with the probability p(bn10η) and Hη is the entropy of binary symbols generated at the state η:
Hη =−p(0|η) log2p(0|η) − (1 − p(0|η)) log2(1− p(0|η)).
The following theorem establishes the relationship between the decomposition tree τ , the (redundancy of) coding techniques used at the nodes, and the resulting redun- dancy R. The theorem also helps to pose some interesting properties of the method, which are stated as Corollaries 1 – 4.
Theorem 3. Let the source generate a message of length n of symbols from an alphabet A = {a0, a1, . . . , am−1}. Let this message be sequentially coded using the binary tree decomposition technique described above. Then the redundancy is defined as
R = ¯nX
η∈Λ
πηρη, (3.5)
1Otherwise, one would come to a (dynamic) Huffman coding technique and there would be no need for binary coding (at least for sources with an entropy larger than 1 bit).
where ¯n and ρη are defined by (3.2), (3.4) and πη = n0η/¯nn can be viewed as a probability of the state η.
Remark: In this formula, ¯n and πη are defined by the decomposition tree, whereas ρη is determined by binary coding techniques, used at the nodes.
Proof. The proof is straightforward. We rewrite (3.5) as follows:
R = ¯nX
η∈Λ
πηρη
= ¯nX
η∈Λ
πη
1 n0η
X
bn0η1 ∈Bn0η
p(bn10η)|ϕb(bn10η)| + Hη
= ¯nX
η∈Λ
πη
1 n0η
X
bn0η1 ∈Bn0η
p(bn10η)|ϕb(bn10η)| − ¯nX
η∈Λ
πηHη
= 1 n
X
η∈Λ
X
bn0η1 ∈Bn0η
p(bn10η)|ϕb(bn10η)| − ¯nX
η∈Λ
πηHη
The first term in the last equality is the average code length and the second defines the entropy of the input source.
Corollary 1. If for ∀η ∈ Λ ρη = ρ, then R = ¯nρ.
Corollary 2. If r-bits registers are used to represent probabilities and an arithmetic coder with registers of size g ≥ r + 2 is used to code binary symbols at the nodes, then R < 2¯n(r + log2e)2−(g−2) + 2/n. (3.6) Proof. It was proven in [46], that the per-symbol redundancy for an m-ary arithmetic coder is upper bounded by the inequality
ρ < m(r + log2e)2−(g−2). (3.7) For binary coding m = 2, and two additional bits are required to terminate the message. This yields (3.6).
Corollary 3. If the decomposition tree is a Huffman tree, r-bits registers are used to represent probabilities and a binary arithmetic coder with registers of size g≥ r + 2 is used to code at the nodes, then
R < (H + 1)(r + log2e)2−(g−2) + 2/n (3.8)
Proof. The proof follows immediately from the fact that for the Huffman tree
¯
n < H + 1. (3.9)
This corollary is essential for coding low entropy sources (H < 1) with large alpha- bets (m 2). In this case, the use of a Huffman tree results in the redundancy of order O(H + 1), independent on the alphabet size, whereas for an m-ary arithmetic coding the redundancy is of order O(m), see (3.7).
Finally, it is always possible to choose such a tree, that ¯n≤ dlog2me.
Corollary 4. If the decomposition tree is such that ¯n ≤ dlog2me, r-bits registers are used to represent probabilities and a binary arithmetic coder with registers of size g ≥ r + 2 is used to code at the nodes, then
R < 2(log2m + 1)(r + log2e)2−(g−2)+ 2/n
Thus, for sources with an alphabet of size m > 4, the upper bound on the coding redundancy using binary decomposition combined with binary arithmetic coding is lower than that of m-ary arithmetic coding (see (3.7)).
3.4 Binary decomposition of FSM sources
An alphabet binarization can be as well used in coding FSM sources. Let an FSM source be defined by the alphabet A, a set of states S = {s}, the initial state s1 and a state transition rule. In this case, each state is assigned a decomposition tree τs ∈ Tm with the corresponding set of states (decision nodes) Λs = {ηs}. In general, the decomposition trees τs can be different (and may be also optimized) for each state.
At each state the source generates a binary string of decisions corresponding to a source symbol. The resulting binary sequence can be modeled as a “double” FSM source with the set of states, which is the product of the sets:
S = Λs× S.
3.5 Binary decomposition and universal coding
Consider a memoryless m-ary source. If the source parameters P (A) = {p(a), a ∈ A}
are not known, then the subsequences of binary decisions can be coded using a universal coding approach. In this case, the decision probabilities can be estimated according to (2.28), where s corresponds to a node η.
The use of a binary decomposition technique does not increase the stochastic com- plexity of the source model, since the number of parameters is retained the same. An m-ary memoryless source and its binary counterpart have m− 1 free parameters (in the latter case this is the number of internal nodes of the decomposition tree). This concerns FSM sources as well, where the number of parameters is (m− 1)|S|.
Thus, binarization of source symbols at least does not make a compression algo- rithm worse in terms universal coding redundancy (2.25) or (2.26) (due to the lack of knowledge on the parameters). Asymptotically this redundancy is the same. Yet, for finite length messages, the redundancy may be even less, resulting in a better compres- sion performance. We will show it in the following example.
Let us assume a memoryless m-ary source, and let the output message un1 of length n > 0 be coded by an ideal m-ary arithmetic coder using adaptive coding probability calculation defined by (2.29) with the parameter vector αa = 12, ∀a ∈ A. Then, the code length is defined as
|ϕ(un1)| = Γ m2 πm2Γ n + m2
Y
a∈A
Γ
ϑ(a|un1) + 1 2
= n ¯H +m− 1
2 log2n + O1(1).
(3.10)
where Γ(·) is a Gamma function, ϑ(a|un1) denotes the symbol counts and H =¯ −X
a∈A
ϑ(a|un1)
n log2 ϑ(a|un1)
n (3.11)
is the empirical entropy of the message.
Let the same message be coded using the binary decomposition technique combined with an ideal binary arithmetic coder. Then the code length is the sum of the codes of binary sub-strings, corresponding to the decomposition tree nodes:
|ϕ(un1)| =X
η∈Λ
|ϕ(bn1η(η))|
=X
η∈Λ
nηH¯η +1
2log2nη + Oη(1)
.
(3.12)
Thus
|ϕ(un1)| =X
η∈Λ
nηH¯η +1
2log2Y
η∈Λ
nη + O2(1) (3.13)
where nη denotes lengths of the binary sub-strings, H¯η =−ϑ(0|un1η)
nη
log2ϑ(0|un1η)
nη − ϑ(1|un1η) nη
log2ϑ(1|un1η) nη
(3.14) is the empirical entropy of the binary sub-strings and
O2(1) =X
η∈Λ
Oη(1). (3.15)
It can be shown that
n ¯H =X
η∈Λ
nηH¯η. (3.16)
The second term in (3.13) has its maximum when
nη = n0 (3.17)
∀η ∈ Λ, i.e., all nη are equal to each other. n0 can be estimated as n0 = n¯n
m < n (3.18)
where ¯n is the average number of binary decision per symbol. Thus, we have 1
2log2Y
η∈Λ
nη ≤ m− 1
2 log2n−m− 1
2 log2 m
¯
n, (3.19)
and thereby
|ϕ(un1)| ≤ n ¯H + m− 1
2 log2n + O3(1), (3.20) where
O3(1) = O2(1)− m− 1
2 log2 m
¯
n. (3.21)
If m is large enough and ¯n is small, then O3(1) may be less than O1(1). Moreover, the inequality (3.19) overestimates the second term in (3.13) due to assumptions (3.17) and (3.18).
Thus, in many practical cases, especially if the probability distribution is non- uniform (skewed), the use of a binary decomposition combined with binary arithmetic coding may result in a shorter code length compared with m-ary arithmetic coding.
Applications of binary
decomposition to source coding
4.1 Introduction
In this chapter, we use the binarization technique to develop methods for efficient coding of parametric sources such as the generalized two-sided geometric distribution (GTSGD). The GTSGD is an extension of the two-sided geometric distribution (TSGD) introduced in [28]. It often occurs in image compression algorithms. The distribution will be introduced in Section 4.2. The advantage of the GTSGD model is a small num- ber of parameters that allows to avoid the context dilution problem in multi-context algorithms (see Chapter 5). Direct implementation of conventional arithmetic coding requires as many coding parameters as the alphabet size, which is usually much larger than the number of the distribution parameters. We will show that the binary decom- position allows for an efficient solution to this problem. Two approaches are described in Section 4.3. Some experimental results are presented in Section 4.4. Finally, in Section 4.5 we use binary decomposition for redundancy analysis of a widely used Rice coding algorithm [38].
4.2 Generalized two-sided geometric distribution
The generalized two-sided geometric distribution (GTSGD) is deduced as a dead-zone quantization of the off-centered continuous Laplacian distribution
f (x) = β
2exp−β|x−ε|,
which was shown to be a good approximation of the distribution of prediction errors and transform coefficients [31, 37]. The distribution is specified by the decay parameter β and the offset ε. The dead-zone quantizer is non-uniform and specified by the zero/non- zero quantization intervals Q0 and Q, respectively. The quantizer and the distribution are depicted on Figure 4.1. We assume that ε < Q0/2, i.e., the distribution center falls into the zero quantization bin. This restriction is justified by the fact, that in practice
25
- 6
- - -
-
-
-
Q0 Q Q
Q Q
ε
f (x) = β
2 exp−β|x−ε|
0
x
. . . .
Figure 4.1. Example of the off-centered Laplacian distribution and the dead-zone quantizer.
for transform coefficients it normally holds, and for context-based prediction schemes, the unit interval containing the center of the distribution can be located by an error feedback loop [60, 68].
The probability distribution of output symbols i∈ Z of the quantizer is defined as
p(i = 0) = Z Q02
−Q02
f (x)dx,
p(i > 0) =
Z Q02 +iQ
Q0
2 +(i−1)Q
f (x)dx,
p(i < 0) =
Z −Q02 +(i+1)Q
−Q02 +iQ
f (x)dx,
and after a little algebra we find
p(i) =
1−1
2θλ2 θγ+ θ−γ
, i = 0 1
2θλ2−γ−1(1− θ)θi, i > 0 1
2θλ2+γ−1(1− θ)θ|i|, i < 0
(4.1)
where θ = exp−βQ, γ = ε/Q and λ = Q0/Q are the new distribution parameters.
Introducing new parameters, we disengage from the Laplacian distribution and the quantizer parameters. The parameter λ controls the probability of the zero symbol, whereas γ and θ define the off-set and the rate of decay of the distribution, respectively.
We shall call the distribution (4.1) the generalized two-sided geometric distribution (GTSGD). It comprises the two-sided geometric distribution (TSGD) proposed in [28]
for modeling prediction residuals in lossless image compression algorithms:
p(i) = (1− θ)θ|i+γ|
θ1−γ + θγ , (4.2)
where θ and γ have the same meaning as in (4.1). The TSGD can be derived from (4.1) by setting
λ = Q0
Q = 2
1 + ln 2− ln(θ1−γ + θγ) ln θ
. (4.3)
4.3 Efficient coding of sources with the GTSGD
Direct use of m-ary arithmetic coding may be inefficient for coding sources with the GTSGD. The main problem is the alphabet size. Although (4.1) assumes an infinite alphabet, in practice it is finite, but quite large (e.g., the potential range of DCT coefficients in the JPEG image compression standard is supposed to be in the range
−2047 . . . 2047 for images with 8 bits per sample [33]). Using arithmetic coding, one must store and update as many parameters as the alphabet size1. Furthermore, such skewed distribution may cause waste of the code space and thereby, reduce the coding efficiency.
The binary decomposition allows to overcome the aforementioned problems and efficiently code sources with a large alphabet and skewed distribution. Using multipli- cation free arithmetic coding, one may also design fast algorithms. An efficient method using binary decomposition of the source alphabet combined with a binary arithmetic coder, was proposed for coding DC and AC coefficients of the DCT in the JPEG image compression standard [33].
In this section, we describe two binarization methods for coding sources with the GTSGD. The methods are optimal in a sense that the number of coding parameters
1Of course, for the source (4.1) one could estimate, store and update only three parameters and calculate the probabilities for the whole alphabet, but this would decrease the coding speed.
