First Edition
U. Chuks
Copyright © 2010 by U. Chuks
Cover design by U. Chuks
Book design by U. Chuks
All rights reserved.
No part of this book may be reproduced in any
form or by any electronic or mechanical means
including information storage and retrieval
systems, without permission in writing from the
author. The only exception is by a reviewer, who
may quote short excerpts in a review.
U. Chuks
Visit my page at
iii
Contents
Table of Contents
Contents ... iii Preface ... vi Chapter 1 ... 1 Introduction ... 11.1 Overview of Digital Image Processing ... 1
1.1.1 Application Areas ... 2
1.2 Digital Image Filtering ... 2
1.2.1 Frequency Domain ... 2
1.2.2 Spatial Domain ... 4
1.3 VHDL Development Environment ... 6
1.3.1 Creating a new project in ModelSim ... 7
1.3.2 Creating a new project in Xilinx ISE ... 14
1.3.3 Image file data in VHDL image processing ... 18
1.3.4 Notes on VHDL for Image Processing ... 20
References... 23
Chapter 2 ... 25
Spatial Filter Hardware Architectures ... 25
2.1 Linear Filter Architectures ... 25
2.1.1 Generic Filter architecture ... 28
2.1.2 Separable Filter architecture ... 30
iv
2.1.4 Quadrant Symmetric Filter architecture ... 34
2.2 Non-linear Filter Architectures ... 35
Summary ... 35 References... 36 Chapter 3 ... 37 Image Reconstruction ... 37 3.1 Image Demosaicking ... 37 3.2 VHDL implementation... 44 3.2.1 Image Selection ... 49 Summary ... 57 References... 57 Chapter 4 ... 59 Image Enhancement ... 59 4.1 Point-based Enhancement ... 60 4.1.1 Logarithm Transform ... 60 4.1.2 Gamma Correction ... 62 4.1.3 Histogram Clipping ... 62 4.2 Local/neighbourhood enhancement ... 64 4.2.1 Unsharp Masking ... 64
4.2.2 Logarithmic local adaptive enhancement ... 65
4.3 Global/Frequency Domain Enhancement ... 65
4.3.1 Homomorphic filter ... 66
4.4 VHDL implementation... 66
Summary ... 68
References... 68
v
Image Edge Detection and Smoothing ... 70
5.1 Image edge detection kernels... 70
5.1.1 Sobel edge filter ... 71
5.1.2 Prewitt edge filter ... 72
5.1.3 High Pass Filter ... 73
5.2 Image Smoothing Filters ... 74
5.2.1 Mean/Averaging filter... 75
5.2.2 Gaussian Lowpass filter ... 75
Summary ... 77
References... 77
Chapter 6 ... 78
Colour Image Conversion... 78
6.1 Additive colour spaces ... 78
6.2 Subtractive Colour spaces ... 79
6.3 Video Colour spaces ... 82
6.4 Non-linear/non-trivial colour spaces ... 91
Summary ... 95
References... 95
Circuit Schematics ... 97
Creating Projects/Files in VHDL Environment ... 106
VHDL Code ... 118
vi
Preface
The relative dearth of books regarding the know-how involved in implementing several algorithms in hardware was the motivating factor in writing this book, which was written for those with a prior understanding of image processing fundamentals who may or may not be familiar with programming environments such as MATLAB and VHDL. Thus, the subject is addressed very early on, bypassing the fundamental theories of image processing, which are better covered in several contemporary books given in the references sections in the chapters of this book.
By delving into the architectural design and implications of the chosen algorithms, the user is familiarized with the necessary tools to realize an algorithm from theory to software to designing hardware architectures.
Though the book does not discuss the vast theoretical mathematical processes underlying image processing, it is hoped that by providing working examples of actual VHDL and MATLAB code and simulation results of the software, that the concepts of practical image processing can be appreciated.
This first edition of this book attempts to provide a working aid to readers who wish to use the VHDL hardware description language for implementing image processing algorithms from software.
1
Chapter 1
Introduction
Digital image processing is an extremely broad and ever expanding discipline as more applications, techniques and products utilize digital image capture in some form or the other. From industrial processes like manufacturing to consumer devices like video games and cameras, etc, image processing chips and algorithms have become ubiquitous in everyday life.
1.1 Overview of Digital Image Processing
Image processing can be performed in certain domains using:
Point (pixel-by-pixel) processing operations. Local /neighbourhood/window mask operations. Global processing operations.
A list of the areas of digital image processing includes but is not limited to:
Image Acquisition and Reconstruction Image Enhancement
Image Restoration
Geometric Transformations and Image Registration Colour Image Processing
Image Compression
Morphological Image Processing Image Segmentation
Object and Pattern Recognition
2
Image Reconstruction, Enhancement and Colour Image Processing and the VHDL implementation of selected algorithms from these areas.
1.1.1 Application Areas
Image Reconstruction and Enhancement techniques are used in digital cameras, photography, TV and computer vision chips.
Colour Image and Video Enhancement is used in digital video, photography, medical imaging, remote sensing and forensic investigation.
Colour Image processing involves colour segmentation, detection, recognition and feature extraction.
1.2 Digital Image Filtering
Digital image filtering is a very powerful and vital area of image processing, with convolution as the fundamental and underlying mathematical operation that underpins the process makes filtering one of the most important and studied topics in digital signal and image processing.
Digital image filtering can be performed in the Frequency, Spatial or Wavelet domain and operating in any of these domains requires a domain transformation or changing the representation of a signal or image into a form in which it is easier to visualize and/or modify the particular aspect of the signal one wishes to analyze, observe or improve upon.
1.2.1 Frequency Domain
Filtering in the frequency domain involves transforming an image into a representation of its spectral components and then using a frequency filter to modify and alter the image
3
by passing a particular frequency and suppressing or eliminating other unwanted frequency components. This frequency transform can involve the famous Fourier Transform or Cosine Transform. Other frequency transforms also exist in the literature but these are the most popular. The (Discrete) Fourier transform is another core component in digital image processing and signal analysis. The transform is built on the premise that complex signals can be formed from fundamental and basic signals when combined together spectrally. For a discrete image function, of M ×N dimensions with spatial coordinates, x and y, the DFT transform is given as;
(1.2.1-1)
And its inverse transform back to the spatial domain is;
(1.2.1-2)
Where is the discrete image function in the frequency domain with frequency coordinates, u and v, and j is the imaginary component. The basic steps involved in frequency domain processing are shown in Figure 1.2.1(i).
Figure 1.2.1(i) - Fundamental steps of frequency domain filtering
Frequency Domain Filter Inverse Fourier Transform Fourier Transform Pre- Processing Post- Processing
4
The frequency domain is more intuitive due to the transformation of the spatial image information to frequency-dependent information. The frequency transformation makes it is easier to analyze image features across a range of frequencies. Figure 1.2.1(ii) illustrates the frequency transformation of the spatial information inherent in an image.
(a) (b)
Figure 1.2.1(ii) – (a) Image in spatial domain (b) Image in frequency domain
1.2.2 Spatial Domain
Spatial domain processing operates on signals in two dimensional space or higher, e.g. grayscale, colour and MRI images. Spatial domain image processing can be point-based, neighbourhood/kernel/mask or global processing operations.
The spatial domain mask filtering involves convolving a small spatial filter kernel or mask around a local region of the image, performing the task repeatedly until the entire image is processed. Linear spatial filtering processes each pixel as a linear combination of the surrounding, adjacent neighbourhood pixels while non-linear spatial filtering uses statistical, set theory or logical if-else operations to process
5
each pixel in an image. Examples include the median and variance filters used in image restoration. Figure 1.2.2(i) show the basics of spatial domain processing where is the input image and is the processed output image. Pre-processing Filter Function Post-processing Io( yx, ) ) , ( yx i I
Figure 1.2.2(i) - Basic steps in spatial domain filtering
Spatial domain filtering is highly favoured in hardware image processing filtering implementations due to the practical feasibility of employing it in real-time industrial processes. Figure 1.2.2(ii) shows the plots of a frequency response of the filter and the spatial domain equivalent for high and low pass filters.
(a) (b)
(c) (d)
Figure 1.2.2(ii) – Low-pass filter in the (a) frequency domain (b) spatial domain and High-pass filter in the (c) frequency domain (d) spatial domain
6
This gives an idea of the span of the spatial domain filter kernels relative to their frequency domain counterpart. Since a lot of the algorithms in this book involve spatial domain filtering techniques and their implementation in hardware description languages (HDLs), emphasis will be placed on spatial domain processing throughout the book.
1.3 VHDL Development Environment
VHDL is one of the languages for describing the behaviour of digital hardware devices and highly complex circuits such as FPGAs, ASICs and CPLDs. In other words, it is called a hardware description language (HDL) and others include ADA and Verilog, which is the other commonly-used HDL. VHDL is preferred because of its open source nature in that it is freely available and has a lot of user input and support helping to improve and develop the language further. There has been three or four language revisions of VHDL since its inception in the 80s, and have varying syntax rules.
Tools for hardware development with VHDL include such popular software such as ModelSim for simulation and Xilinx ISE tools and Leonardo Spectrum for complete circuit design and development. With software environments like MathWorks MATLAB and Microsoft Visual Studio, image processing algorithms and theory can now be much more easily implemented and verified in software before being rolled out into physical, digital hardware.
We will be using the Xilinx software and ModelSim software for Xilinx devices for the purposes of this book.
7
1.3.1 Creating a new project in ModelSim
Before proceeding, ModelSim software from Mentor Graphics must be installed and enabled. Free ModelSim software can be downloaded from internet sites like Xilinx website or other sources. The one used for this example is a much earlier version of ModelSim (version 6.0a) tailored for Xilinx devices.
Once ModelSim is installed, run it and the window like the one in Figure 1.3.1(i) should appear.
Figure 1.3.1(i) – ModelSim starting window
Close the welcome page and click on File, select New -> Project as shown in Figure 1.3.1(ii).
Click on the Project option and a dialog box appears as shown in Figure 1.3.1(iii). You can then enter the project name. However we would select an appropriate location to
8
store all project files to have a more organized work folder. Thus, click on Browse and the dialog box shown in Figure 1.3.1(iv) appears. Now we can navigate to an appropriate folder or create one if it doesn‟t exist. In this case, a previously created folder called „colour space converters‟ was created to store the project files. Clicking „OK‟ returns us to the „Create a New Project‟ dialog box and now we name the project as „Colour space converters‟ and click „OK‟.
9
A small window appears for us to add a new or existing file as shown in Appendix B, Figure B1.
Since we would like to add a new file for illustrative purposes, we create a file called „example_file‟ as in Figure B3 and it appears on the left hand side workspace as depicted in Figure B4.
Then we add existing files by clicking the „Add Existing File‟ and navigate to the relevant files and select them as shown in Figure B5. They now appear alongside the newly created file as shown in Figure B6.
The rest of the process is easy to follow. For further instruction on doing this, refer to Appendix B or the Xilinx sources listed at the end of the chapter.
Now these files can be compiled before simulation as shown in the subsequent figures.
Successful compilation is indicated by messages in green colours while a failed compilation messages are in red and will indicate the errors and the location of those errors like all smart debugging editors for software code development. Any errors are located and corrected and the files recompiled until there are no more syntax errors.
10
Figure 1.3.1(iii) – Creating a new project
Once there are no more errors, the simulation of the files can begin. Clicking on the simulation tab will open up a window to select the files to be simulated. However, you must create a test bench file for simulation before running any simulation. A test bench file is simply a test file to evaluate your designed system to verify its correct functionality.
You can choose to add several more windows to view the ports and signals in your design.
11
Figure 1.3.1(iv) – Changing directory for new project
The newly created file is empty upon inspection, thus we have to add some code to the blank file. We start with including and importing the standard IEEE libraries needed as shown in Figure 1.3.1(v) at the top of the blank file. library IEEE;
use IEEE.std_logic_1164.all; use IEEE.std_logic_arith.all;
12
The “IEEE.std_logic_1164” and the
“IEEE.std_logic_arith” are the standard logic and the standard logic arithmetic libraries, which are the minimum libraries needed for any VHDL logic design since they contain all the necessary logic functions.
With that done, the next step would be to add the architecture of the system we would like to describe in this example file. Thus, the block diagram for the design we are going to implement in VHDL is shown in Figure 1.3.1(vi).
Figure 1.3.1(vi) – Top level system description of example_file This leads to the top level architecture description in VHDL code shown in Figure 1.3.1(vii).
----TOP SYSTEM LEVEL DESCRIPTION---
entity example_file is
port ( ---the collection of all input and output ports in top level
Clk : in std_logic; ---clock for synchronization rst : in std_logic; ---reset signals for new data input_port : in bit; ---input port
output_port : out bit ---output port );
end example_file;
Figure 1.3.1(vii) – VHDL code for black box description of example_file rst
input_port
example_file
clk
13
The code in Figure 1.3.1(vii) is the textual or code description of the black box diagram shown in Figure 1.3.1(vi).
The next step is to detail the actual operation of the system and the relationship between the input and output ports and this operation of the system is shown in the VHDL code in Figure 1.3.1(viii).
---architecture and behaviour of TOP SYSTEM LEVEL DESCRIPTION in more detail
architecture behaviour of example_file is ---list signals which connect input to output ports here
---for example
signal intermediate_port : bit := '0'; --initialize to zero
begin ---start
process(clk, rst) --process which is triggered by clock or reset pin
begin
if rst = '0' then --reset all output ports intermediate_port <= '0'; --initialize output_port <= '0'; --initialize elsif clk'event and clk = '1' then --operate on rising edge of clock
intermediate_port <= not(input_port); --logical inverter
output_port <= intermediate_port or
input_port; --logical or operation
end if;
end process; --self-explanatory
end behaviour; --end of architectural behaviour
14
The first line of code in Figure 1.3.1(viii) defines the beginning of the behavioural level of the architecture. The next line defines a signal or wire that will be used in connecting the input port to the output port. It has been defined as a single bit and initialized to zero.
The next line indicates the beginning of a triggered process that responds to both the clock and reset signals.
The if…then…else…then statements indicate what actions and statements to trigger when the stated conditions are met.
The actual logical operation starts at the rising edge of the clock and the signal takes on the value from the input port and inverts it while the output port performs the logical „or‟ operation on the inverted and non-inverted signals to produce the output value. Though this is an elaborate circuit design for a simple inverter operation, it was added to illustrate several aspects that will be recurring themes throughout the work discussed in the book.
1.3.2 Creating a new project in Xilinx ISE
Like the ModelSim software, the software for evaluating VHDL designs in FPGA devices can be downloaded for free from FPGA Vendors like Leonardo Spectrum for Altera and Actel FPGAs or the Xilinx Project Navigator software from Xilinx. The Xilinx ISE version used in this book is 7.1.
Once the software has been fully installed, we can then begin, so by opening the program, we get a welcome screen, just like that when we launched ModelSim.
15
Creating a project in the Xilinx ISE is similar to the process in ModelSim., however one would have to select the specific FPGA device for which the design is to be loaded. This is because the design must be physically mapped onto a physical device and the ISE software is comprised of special, complicated algorithms that emulate the actual hardware device to ensure that the design is safe and error-free before being downloaded to an actual device. This saves on costly errors and damage to the device by incorrectly routed pins when designing for large and expensive devices like ASICs.
A brief introduction to creating a project in Xilinx is shown in Figure 1.3.2(i) – 1.3.2(iv).
16
We then click „OK‟ on the welcome dialog box to access the project workspace. Then click on File, select New Project as shown in Figure 1.3.2(ii) and enter a new name for the project as shown in Figure 1.3.2(iii). Then click „Next‟ and the next window shown in Figure 1.3.2(iv) prompts you to select the FPGA hardware device family your final design is going to be implemented in. We select the Xilinx Spartan 3 FPGA chip which is indicated by the chip number xc3s200 and the package is ft256 and the speed grade is -4. This device will be referred to as 3s200ft256-4 in the Project Navigator.
We leave all the other options as they are since we will be using the ModelSim simulator and use the VHDL language for most of the work and only implementing the final design after correct simulation and verification.
Depending on the device you are implementing your design on, the device family name will be different. However, the cost of the free software means that you do not have access to all the FPGA devices in every available device family in the software‟s database and thus will not be able to generate a programming file to be downloaded to an actual FPGA.
The design process from theoretical algorithm description to circuit development and flashing to an FPGA device is a non-linear exercise as the design may need to be optimized and/or modified depending on the design constraints of the project.
17
Figure 1.3.2(ii) – Creating a new project in Xilinx Project Navigator
18
Figure 1.3.2(iv) – Selecting a Xilinx FPGA target device
Clicking Next to the next set of options allows you to add HDL source files, similar to ModelSim. The user can add them from here or just click through to create the project and then add the files manually like in ModelSim.
1.3.3 Image file data in VHDL image processing
Figure 1.3.3 shows an image in the form of a text file, which will be read using the textio library in VHDL. A software program was written to convert image files to text in order to process them. The images can be converted to any numerical type including binary, hexadecimal (to save space). Integers were chosen for easy readability and debugging and for illustration of the concepts. After doing this, another software program is written to convert the text files back to images to be viewed.
19
Writing MATLAB code is the easiest and quickest way of doing this when working with VHDL. MATLAB also enables fast and easy prototyping of algorithms without re-inventing the wheel and being force to write each and every function needed to perform standard operations, especially image processing algorithms. This was why it was chosen over the .NET environment.
Coding in VHDL is a much different experience than coding with MATLAB, C++ or JAVA since it is describing hardware circuits, which have to be designed as circuits rather than simply software programs.
VHDL makes it much easier to describe highly complex circuits that would be impractical to design with basic logic gates and it infers the fundamental logical behaviour based on the nature of the operation you describe within the code. In a sense, it is similar to the Unified Modeling Language (UML) used to design and model large and complex object-oriented software algorithms and systems in software engineering.
SIMULINK in MATLAB is also similar to this and new tools have been developed to allow designers with little to know knowledge of VHDL to work with MATLAB and VHDL code. However, the costs of these tools are quite prohibitive for the average designer with a small budget.
FPGA system development requires a reasonable amount of financial investment and the actual prototype hardware chip cost can be quite considerable in addition to the software tools needed to support the hardware. Thus, with
20
these free tools and a little time spent on learning VHDL, designing new systems becomes much more fulfilling and gives the coder the chance to really learn about how the code and the system they are trying to build is going to work on a macro and micro level. Also, extensive periods debugging VHDL code will definitely make the coder a much better programmer because of the experience.
Figure 1.3.3 – image as a text file to be read into VHDL testbench
1.3.4 Notes on VHDL for Image Processing
Most users of this book probably have had some exposure to programming or at least have heard of programming languages and packages like C++, JAVA, C, C#, Visual Basic, MATLAB, etc. But fewer people are aware of languages like VHDL and other HDLs like Verilog and ADA, which make it much easier to design larger and more complex circuits for digital hardware chips like ASICs, FPGAs, and CPLDs used in highly sophisticated systems and devices.
21
When using these fourth generation languages like C# and MATLAB, writing programs to perform mathematical tasks and operations is much easier and users can make use of existing libraries to build larger scale systems that perform more complex mathematical computations without thinking much about them.
However, with languages like VHDL, performing certain mathematical computations like statistical calculations or even divisions require careful system design and planning if the end product is to realize a fully synthesizable circuit for downloading to an FPGA. In order words, floating point calculations in VHDL for FPGAs is a painful and difficult task for the uninitiated and those without developer and design resources. Some hardware vendors have developed their own specialized floating point cores but these come at a premium cost and are not for the average hardware design hobbyist. Floating point calculations take up a lot of system resources and along with operations like divisions, especially when calculating non-multiples of 2. Thus, most experienced designers prefer to work with fixed-point mathematical calculations.
For example, if we choose to write a program to calculate the logarithm, cosine or exponential of signal values, this is usually taken care of in software implementation by calling a log, cosine or exponential function from the inbuilt library without even being aware of the algorithm behind the function. This is not the case with VHDL or hardware implementation. Though it is vital to note that VHDL has libraries for all these non-linear functions, the freely available functions are not synthesizable. This means that they cannot be realized in digital hardware and thus
22
hardware design engineers must devise efficient architectures for these algorithms or purchase hardware IP cores developed by FPGA vendors before they can be implement them on an FPGA.
The first obvious route to building these type of functions is to create a look-up-table (LUT) consisting of pre-calculated entries in addressable memory (ROM) which can then accessed for a defined range of values. However, the size of the LUT can expand to unmanageable proportions and render the entire system inefficient, cumbersome and wasteful. Thus, a better approach would involve a mixture of some pre-computed values and the calculation of other values to reduce the memory size and increase efficiency. Thus, the LUT is a constant recurring theme in hardware design involving certain systems that perform intensive mathematical computation and signal processing.
Usually, when a non-linear component is an essential part of an algorithm, the LUT becomes an alternative to implementing such crucial part of the algorithm or an alternative algorithm may have to be devised in accordance with error trade-off curves. This is the standard theme of research papers and journals on digital logic circuits.
Newer and more expensive FPGAs now have a soft core chip built into them, enabling the designer the flexibility of apportioning soft computing tasks to the PC chip on the FPGA while devoting more appropriate device resources to architectural demands. However the other challenge of real-time reconfigurable computing and linking both the soft core and the hard core aspects of the system to work in tandem comes into play.
23
Most of the images used in this book are well known in the image processing community and were obtained from the University of South Carolina Signal and Image Processing Institute website and others from relevant research papers and online repositories.
References
R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2 ed.: Prentice Hall, 2002.
R. C. Gonzalez, R. E. Woods, and S. L. Eddins, Digital Image Processing Using MATLAB: Prentice Hall, 2004.
W. K. Pratt, Digital Image Processing, 4 ed.: Wiley-Interscience, 2007.
U. Nnolim, “FPGA Architectures for Logarithmic Colour Image Processing”, Ph.D. thesis, University of Kent at Canterbury, Canterbury-Kent, 2009.
MathWorks, "Image Processing Toolbox 6 User's Guide for use with MATLAB," The Mathworks, 2008, pp. 285 - 288.
[6] Mathworks, "Designing Linear Filters in the Frequency Domain," in Image Processing Toolbox for use with MATLAB, T. Mathworks, Ed.: The Mathworks, 2008.
Mathworks, "Filter Design Toolbox 4.5," 2009.
Weber, "The USC-SIPI Image Database," University of South Carolina Signal and Image Processing Institute (USC-SIPI), 1981.
Zuloaga, J. L. Martín, U. Bidarte, and J. A. Ezquerra, "VHDL test bench for digital image processing systems using a new image format."
Cyliax, "The FPGA Tour: Learning the ropes," in Circuit Cellar online, 1999.
T. Johnston, K. T. Gribbon, and D. G. Bailey, "Implementing Image Processing Algorithms on FPGAs," in Proceedings of the Eleventh Electronics New Zealand Conference (ENZCon‟04), Palmerston North, 2004, pp. 118 - 123.
EETimes, "PLDs/FPGAs," 2009.
Digilent, "http://www.digilentinc.com," 2009.
E. R. Davies, Machine Vision: Theory, Algorithms, Practicalities 3rd ed.: Morgan Kaufmann Publishers, 2005.
24
www.xilinx.com, "FPGA Design Flow Overview (ISE Help)." vol. 2008: Xilinx, 2005.
25
Chapter 2
Spatial Filter Hardware Architectures
Prior to the implementation of the various filters, it is necessary to lay the groundwork for the design of spatial filter hardware architectures in VHDL.
2.1 Linear Filter Architectures
Using spatial filter kernels for image filtering applications in hardware systems has been a standard route for many hardware design engineers. As a result, various architectures in the spatial domain exist in company technical reports, academic journals and conferences papers dedicated to digital FPGA hardware-based image processing. This is not surprising because of the myriad of image processing applications that incorporate image filtering techniques.
Such applications include but are not limited to image contrast enhancement/sharpening, demosaicking, restoration/noise removal/deblurring, edge detection, pattern recognition, segmentation, inpainting, etc.
Several authors have published papers involving implementing a myriad of algorithms involving spatial filtering hardware architectures for FPGA platforms performing different tasks or used as add-ons for even more complex and sophisticated processing operations..
26
A sample of application areas in industrial processes include the detection of structural defects in manufactured products using real-time imaging and edge detection techniques to remove damaged products from the assembly line.
Though frequency (Fourier Transform) domain filtering may be faster for larger images and optical processes, spatial filtering using relatively small kernels and make several of these processes feasible for physical, real-time applications and reduce computational costs and resources in FPGA digital hardware systems.
Figure 2.1(i) shows one of the essential components of a spatial domain filter, which is a window generator for a 5 x 5 kernel for evaluating the local region of the image.
Line In 1 Line In 2 Line In 5 FF FF FF Line In 3 Line In 4 FF FF FF FF FF FF Line Out 1 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF Line Out 2 Line Out 3 Line Out 4 Line Out 5
Figure 2.1(i) – 5×5 window generator hardware architecture The boxes represent the flip flops (FF) or delay elements with each box providing one delay. In digital signal processing notation, a flip flop is represented in the z-domain by and in the discrete time domain as ,
27
where x would be the delayed signal. The data comes in from the left hand side of the unit and each line is delayed by 5 cycles. For a 3 x 3 kernel, there would be three lines and each would be delayed by 3 cycles.
Figure 2.1(ii) shows the line buffer array unit which consists of long shift registers composed of several flip flops. Each line buffer is set to the length of one row of the image. Thus, for a 128 x 128 greyscale image with 8 bits per pixel, each line buffer would be 128 wide and 8 bits deep.
Line Buffer1 Line Buffer2 Line Buffer5 Line out1 Line out2 Line out5 Data_in
Line Buffer3 Line out3
Line Buffer4 Line out4
Figure 2.1(ii) – Line buffer array hardware architecture
The rest of the architecture would include adders, dividers, and multipliers or look up tables. These are not shown as they are much easier to understand and implement.
The main components of the spatial domain architectures are the window generator and line delay elements. The delay elements can be built from First in First out (FIFO) or shift register components for the line buffers.
28
The architecture of the processing elements is heavily determined by the mathematical properties of the filter kernels. For instance the symmetric or separable nature of certain kernels is incorporated in the hardware design to reduce multiply-accumulate operations. There are mainly three kinds of filter kernels, namely symmetric, separable-symmetric and non-separable, non-separable-symmetric kernels. To understand the need for this clarification, it is necessary to discuss the growth in mathematical operations of image processing algorithms implemented in digital hardware.
2.1.1 Generic Filter architecture
In the standard spatial filter architectures, the filter kernel is defined as is and each coefficient of the defined kernel has its own dedicated multiplier and corresponding image window coefficient. Thus, this architecture is flexible for a particular defined size of kernel and any combination of coefficient values can be loaded to this architecture without modifying the architecture in any way. However, this architecture is inefficient when a set of coefficients in the filter have the same values and redundancy grows as the number of matching coefficients increases. It also becomes computationally complex as filter kernel size increases since more processing elements will be needed to perform the full operation on a similarly sized image window. The utility of the filter is limited to small kernel sizes ranging from 3×3 to about 9×9 dimensions. Beyond this, the definition and instantiation of the architecture and its coefficients become unwieldy, especially in digital hardware description languages used to program the hardware devices. Figure 2.1.1 depicts an example of generic 5×5 filter kernel architecture.
29 Line Buffer Line Buffer Line Buffer Data_in FF FF FF Line Buffer Line Buffer FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF Data_out × × × × × c20 c21 c22 c23 c24 ∑ Data_out × × × × × c5 c6 c7 c8 c9 ∑ Data_out × × × × × c10 c11 c12 c13 c14 ∑ Data_out × × × × × c15 c16 c17 c18 c19 ∑ Data_out × × × × × c0 c1 c2 c3 c4 ∑
Figure 2.1.1 – Generic 5×5 spatial filter hardware architecture The 25 filter coefficients range from c0 to c24 and are multiplied with the values stored in the window generator grid made up of flip flops (FF). These coefficients are weights, which determine the extent of the contribution of
30
the image pixels in the final convolution output. The partial products are then summed in the adder blocks. Not shown in the diagram is another adder block to sum all the five sums of products. The final sum is divided by a constant value, which is usually defined as a multiple of 2 for good digital design practice.
2.1.2 Separable Filter architecture
The separable filter kernel architectures are much more computationally efficient where applicable. However, these are more suited to low-pass filtering using Gaussian kernels (which have the separability property). The architecture reduces a two dimensional N × N sized filter kernel to two, one dimensional filters of length N. Thus a one-dimensional convolution operation (which is much easier than 2-D convolution) is performed followed by multiplication operations. The savings on multiply-accumulate operations as a result in the reduction in the number of processing elements demanded by the architecture can really be truly appreciated when designing very large filter convolution kernel sizes. Due to the fact that spatial domain convolution is more computationally efficient for small filter kernel sizes, separable spatial filter kernels further increase this efficiency (especially for large kernels built as with a generic filter architecture implementation).
Figure 2.1.2 depicts an example of separable filter kernel architecture for a 5 × 5 spatial filter now reduced to 5 since the row and the column filter coefficients are the same with one 1-D filter being the transpose of the other.
31
32
Observing the diagram in Figure 2.1.2, it can be seen that the number of processing elements and filter coefficients have been dramatically reduced in this filter architecture. For example, the 25 coefficients in the generic filter architecture have been reduced to just 5 coefficients which are reused.
2.1.3 Symmetric Filter Kernel architecture
Symmetric filter kernel architectures are more suited to high-pass and high-frequency emphasis (boost filtering) operations with equal weights and reduce the number of processing elements, thereby reducing the number of multiply-accumulate operations. A set of pixels in the image window of interest are added together and then the sum is multiplied by the corresponding coefficient, which has the same value for those particular pixels in their respective, corresponding locations. Figure 2.1.3(i) shows a Gaussian symmetric high-pass filter generated using the windowing method while Figure 2.1.3(ii) depicts an example of symmetric filter kernel architecture
Figure 2.1.3(i) – Frequency domain response of symmetric Gaussian high-pass filter obtained from spatial domain symmetric Gaussian with windowing method
33
34
2.1.4 Quadrant Symmetric Filter architecture
The quadrant symmetric filter is basically one quadrant (or a quarter) of a circular symmetric filter kernel and rotated 360 degrees. The hardware architecture is very efficient since it occupies a quarter of the space normally used for a full filter kernel.
To summarize the discussion of spatial filter hardware architectures, it is necessary to present a comparison of the savings of hardware resources with regards to reduced multiply-accumulate operations.
For an N × N spatial filter kernel, N × N multiplications and (N × N)-1, additions are required. For example, for a 3 × 3 filter, 9 multiplications and 8 additions are needed for each output pixel calculation, while for a 9×9 filter, 81 multiplications and 80 additions are needed per output pixel computation.
Since multiplications are costly in terms of hardware, designs are geared towards reducing the number of multiplication operations or eliminating them entirely.
Table 2.1.4 gives a summary of the number of multiplication and addition operations per image pixel required for varying filter kernel sizes using different filter architectures.
35
Kerne
l size (GFKA) */pixel pixel +/ (GFK A) */ pixel (SFK A) +/pixe l (SFK A) */ pixel (Sym FKA) +/ pixel (Sym FKA) 3×3 9 8 6 4 4/3 8 5×5 25 24 10 8 6/5 24 7×7 49 48 14 12 8/7 48 9×9 81 80 18 16 10/9 80 13×13 169 168 26 24 14/13 168 27×27 729 728 54 52 28/27 728 31×31 961 960 62 60 32/31 960
Table 2.1.4 – MAC operations and filter kernel size and type
KEY
*/pixel – Multiplications per pixel +/pixel – Additions per pixel
GFKA – Generic Filter Kernel Architecture SFKA – Separable Filter Kernel Architecture
Sym FKA – Circular Symmetric Filter Kernel Architecture
2.2 Non-linear Filter Architectures
The nature of non-linear filter architectures is more complex than that of linear filters and depends on the algorithm or order statistics used in the algorithm. Since most of the algorithms covered in this book involve linear filtering, we focus more on linear spatial domain filtering.
Summary
In this section, we discussed several linear spatial filter hardware architectures used for implementing algorithms in FPGAs using VHDLs and analyzed the cost savings of each architecture with regards to use of processing elements in hardware.
36
References
U. Nnolim, “FPGA Architectures for Logarithmic Colour Image Processing”, Ph.D. thesis, University of Kent at Canterbury, Canterbury-Kent, 2009.
Cyliax, "The FPGA Tour: Learning the ropes," in Circuit Cellar online, 1999.
E. Nelson, "Implementation of Image Processing Algorithms on FPGA Hardware," in Department of Electrical Engineering. vol. Master of Science Nashville, TN: Vanderbilt University, 2000, p. 86.
T. Johnston, K. T. Gribbon, and D. G. Bailey, "Implementing Image Processing Algorithms on FPGAs," in Proceedings of the Eleventh Electronics New Zealand Conference (ENZCon‟04), Palmerston North, 2004, pp. 118 - 123.
S. Saponara, L. Fanucci, S. Marsi, G. Ramponi, D. Kammler, and E. M. Witte, "Application-Specific Instruction-Set Processor for Retinex-Like Image and Video Processing," IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 54, pp. 596 - 600, July 2007.
EETimes, "PLDs/FPGAs," 2009.
Google, "Google Directory," in Manufacturers, 2009. Digilent, "http://www.digilentinc.com," 2009.
E. R. Davies, Machine Vision: Theory, Algorithms, Practicalities 3rd ed.: Morgan Kaufmann Publishers, 2005.
Xilinx, "XST User Guide ": http://www.xilinx.com, 2008.
www.xilinx.com, "FPGA Design Flow Overview (ISE Help)." vol. 2008: Xilinx, 2005.
Mathworks, "Designing Linear Filters in the Frequency Domain," in Image Processing Toolbox for use with MATLAB, T. Mathworks, Ed.: The Mathworks, 2008.
37
Chapter 3
Image Reconstruction
The four stages of image retrieval from camera sensor acquisition to display device comprise of Demosaicking, White/Colour Balancing, Gamma Correction and Histogram Clipping. The process of interest in this chapter is the demosaicking stage and the VHDL implementation of the demosaicking algorithm will also be described. The steps of colour image acquisition from the colour filter array are shown in Figure 3.
Figure 3 – Image acquisition process from camera sensor
3.1 Image Demosaicking
The process of demosaicking attempts to reconstruct a full colour image from incomplete sampled colour data from an image sensor overlaid with a colour filter array (CFA) using interpolation techniques.
The Bayer array is the common type of colour filter array used in colour sampling for image acquisition. The other methods of colour image sampling are the Tri-filter, and Fovean sensor. References to these other methods are listed at the end of the chapter.
Before we delve deeper into the mechanics of demosaicking, it is necessary to describe the Bayer filter
Gamma Correction Histogram Clipping Colour Balancing Demosaicking
38
array. This grid system involves a CCD or CMOS sensor chip with M columns and N rows. A colour filter is attached to the sensor in a certain pattern. For example, the colour filters could be arranged in a particular pattern as shown by the Bayer Colour Filter Array architecture shown in Figure 3.1(i). G B G B G R G R G G B G B R G R G G B G B R G R G
Figure 3.1(i) – Bayer Colour Filter Array configuration
Where R, G, B stands for the red, green and blue colour filters respectively and the sensor chip produces an M × N array. There are two green pixels for every red and blue pixel in a 2x2 grid because the CFAs are designed to suit the human sensitivity to green light.
The demosaicking process involves splitting a colour image into its separate colour channels and filtering with an interpolating filter. The final convolution results from each channel are recombined to produce the demosaicked image.
39
The equation for the basic linear interpolation demosaicking algorithm is shown for one image channel of an RGB colour image in (3.1-1 to 3.1-5). (3.1-1) (3.1-2) (3.1-3) Yielding (3.1-4)
Expressing the input image as a function of the output image gives the expression:
1+ (3.1-5) Where and are the original, interpolated stage 1 and 2 images respectively, while is the demosaicked output
image and and are interpolation kernels usually consisting of an arrangement of ones and zeros. In the case of this implementation, and are 3 x 3 spatial domain kernels defined as;
and respectively.
Note the redundant summation of and with , the
original image.
Keeping in mind that this is for one channel of an RGB colour image, this process can be performed on the R and
40
B channels modified for the G channel as will be explained further in the following subsections.
The system level diagram for the process for an RGB colour image is a shown in Figure 3.1(ii):
Figure 3.1(ii) – Image Demosaicking process
In the diagram, the convolution involves an interpolation process in addition to redundant summation
Some example images have been demosaicked to illustrate the results. The first example is the image shown on the left hand side in Figure 3.1(iii), which needs to demosaicked. More examples of demosaicking are shown in Figure 3.1(v) and 3.1(vi).
(a) (b)
Figure 3.1(iii) – (a) Original undersampled RGB image overlaid with bayer colour filter array and (b) demosaicked image
Convolution Interpolatio n Redundant Summation R G B G’ R ’ B ’ ’ Demosaicking
41
(a) (b)
Figure 3.1(iv) – (a) Original undersampled R,G and B channels (b) Interpolated R, G and B channels
42
The images in Figure 3.1(iv) show the gaps in the image channel samples. The checkerboard pattern indicates the loss of colours in between colour pixels by black spaces/pixels in each channel.
A checkerboard filter kernel is generated and convolved with the images in Figure 3.1(iv)-(a) to produce the interpolated images in Figure 3.1(iv)-(b). As can be seen, most of the holes or black pixels have been filled. The images in Figure 3.1(iv)-(b) can be filtered again with checkerboard filters to eliminate all the lines seen in the blue and red channels.
The reason why the green channel is interpolated in one pass is that there are two green pixels for every red and green pixel thus the green channel provides the strongest contribution in each 2 x 2 grid of the array.
It is important to note that there are various demosaicking algorithms and they include Pixel Binning/Doubling, Nearest Neighbour, Bilinear, Smooth Hue Transition, Edge-sensing Bilinear, Relative Edge-sensing Bilinear, Edge-sensing Bilinear 2, Variable Number Gradients, Pattern Recognition interpolation methods. For more information about these methods, consult the sources listed at the end of the chapter. Some comparisons between some of the methods are made using energy images in Figure 3.1(vii).
This is by no means an exhaustive list but indicates that demosaicking is a very important and broad field as evidenced by the volume of published literature, which can be found in several research conference papers and journals.
43
(a) (b) Figure 3.1v) – (a) Image with Bayer pattern (b) Demosaicked image
(a) (b) (c)
(d) (e) (f)
Figure 3.1(vi) – (a) Image combined with Bayer array pattern and demosaicked image using (b) bilinear interpolation (c) Original image (d) demosaicked using bilinear 2 and (e) high quality bilinear (f) Gaussian-laplacian method
It is important to note that current modern digital cameras have the ability to store digital images in raw format, which
44
enables users to accurately demosaick images using software without restricting them to the camera‟s hardware.
(a) (b) (c)
(d) (e) (f)
Figure 3.1(vii) – Energy images calculated using Sobel kernel operators for (a) Original Image (b) combined with Bayer array pattern and demoisaicked image using (c) and (d) bilinear interpolation, (e) Gaussian smoothing with Laplacian and (f) Pixel doubling
3.2 VHDL implementation
In this section, the VHDL implementation of the linear interpolation algorithm used in the demosaicking of RGB colour images will be discussed. The first part of the chapter dealt with the software implementation using MATLAB as the prototyping platform.
Using MATLAB, the implementation was quite trivial, however in the hardware domain, the VHDL implementation of a synthesizable digital circuit for the demosaicking algorithm is going to be a lot more involved as we will discover.
45
Prior to coding in VHDL, the first step is to understand the dataflow and to devise the architecture for the algorithm. A rough start would be to draw a system level diagram that would include all the major processing blocks of the algorithm. A top level system diagram is shown in Fig. 3.2(i).
Figure 3.2(i) –Black Box system top level description of demosaicking This is the black box system specification for this demosaicking algorithm. The next step is to go down a level into the demosaicking box to add more detail to the system. Figure 3.2(ii) shows the system level description of the first interpolation stage of the demosaicking algorithm for the R channel.
Figure 3.2(ii) – System level 1 description showing first interpolation stage of R channel using demosaicking algorithm
The R channel is convolved with a linear spatial filter mask as specified in the previous section used in the MATLAB implementation. The convolved R channel or Rc is then summed with the original R channel to produce an interpolated R channel, Rs. The channel, Rs is then passed on to the second interpolation stage shown in Figure 3.2(iii).
Linear Spatial Filter1
Σ
R Rs Rc Interpolation Stage 1 Demosaicking R G B R’ G’ B’46
In this stage, the Rs channel is then convolved with another linear spatial filter mask, to produce a new signal, Rcs, which is subsequently summed with the original R channel and the Rs output channel from the first interpolation stage. This produces the final interpolated channel, R‟ shown as the output in Figure 3.2(iii).
Figure 3.2(iii) – System level 1 description showing second interpolation stage of R channel using demosaicking algorithm
The block diagrams shown in the Figure 3.2(ii) and (iii) can also be used for the B channel. For the G channel, only the first stage of the interpolation is needed as shown in the original algorithm equations. Thus, the system level description for G is as shown in Figure 3.2(iv).
Figure 3.2(iv) – System level 1 description showing interpolation stage of G channel using demosaicking algorithm
The system design can also be done in SIMULINK, which is the visual system description component of MATLAB. The
Linear Spatial Filter2
Σ
Rs Rc s R Interpolation Stage 2 R’G Spatial Filter1 Linear
Σ
G’ Gc
47
complete circuit would look that shown in Figure 3.2(v).
Figure 3.2(v) – SIMULINK System description of linear interpolation demosaicking algorithm
The diagram designed in SIMULINK shown in Figure 3.2(v) is the system level architecture of the demosaicking algorithm with the major processing blocks.
The next step is to develop and design the crucial inner components of the major processing blocks. Based on the mathematical expression for the algorithm, we know that the system will incorporate 3x3 spatial filters and adders. This leads to the design specification for the spatial filter which is the most crucial component of this algorithm.
Several spatial filter architectures exist in research literature with various modifications and specifications depending on the nature of the desired filter. These basic architectures were discussed in Chapter 2 and include the generic form, separable, symmetric and separable symmetric filters. In this section, we choose the generic 3 x 3 filter architecture using long shift registers for the line buffers to the filter instead of FIFOs. We remember that hardware spatial filter architecture comprises a window generator, pixel counter and line buffers, shift registers, flip flops, adders and
Video Viewer R G B Video Viewer dc168_lenna_bayer.png R G B Image From File
Bs B B_prime interp_filter_b2 Embedded MATLAB Function5 Rs R R_prime interp_filter_r2 Embedded MATLAB Function3 B Bs B1 interp_filter_b Embedded MATLAB Function2 G interp_filter_g G_prime Embedded MATLAB Function1 R Rs R1 interp_filter_r Embedded MATLAB Function
48
multipliers. Building on the spatial filter architectures discussed in Chapter 2, all that needs to be modified in the filter architecture are the coefficients for the filter and the divider settings. Skeleton VHDL codes, which can be modified for this design can be found in the Appendices. A brief snippet of the VHDL code used in constructing the interpolation step for the R channel is shown in Figure 3.2(vi). The top part of the code in Figure 3.2(vi) includes the instantiations of the necessary libraries and packages.
Figure 3.2(vi) – VHDL code snippet for specifying interpolation filter for R channel
49
Figure 3.2(vii) – Visual system level description of the VHDL code snippet for specifying interpolation filter for R channel
The component specification of the “interp_mask_5x5_512” part in the VHDL code shown in Figure 3.2(i) is embedded within the system level description of the interp_filter_r system as described in Figure 3.2(ii) – 3.2(iii).
3.2.1 Image Selection
Now we select the image we want to process so for the convenience, we choose the Lena image overlaid over a CFA array as shown in Figure 3.2.1(i) The criteria for choosing this image includes the familiarity of this image to the image processing community and also because it is a square image (256 x 256), which makes it easier to specify in the hardware filter without having to pad the image or add extra pixels.
Figure 3.2.1(i) – Original image to be demosaicked
Interp_filter_r top_clk top_rst dat_in dat_out D_out_valid
50
Based on what was discussed about demosaicking, we know that the easiest channel to demosaick would be the green channel since there are two green pixels for every red and blue pixel in a 2 x 2 CFA array, thus only one interpolation pass is required. Thus we will discuss the green channel last.
(a) (b)
(c) (d)
Figure 3.2.1(ii) – Demosaicked R image channel from left to right (software simulation) (a) original R channel, (b) filtered channel, Rc, from first stage interpolation (c) filtered channel, Rcs, from second stage interpolation (d) demosaicked image
51
In Figure 3.2.1(ii), we can observe the intermediate interpolation results of the spatial filter from left to right. Red image is the original red channel, (R) of the image in Figure 3.2.1(i). Red1 image is the interpolated image from the first stage or Rc from the diagram in Figure 3.1.2(i). Red2 image is the second interpolated image (Rcs) from Figure 3.1.2(iii) while (d) is the final demosaicked R channel, R‟.
The diagrams shown in Figure 3.2.1(iii), are the image results obtained from both software (a) and the hardware simulation (b). The results show no visually perceptible difference, thus the hardware filter scheme was implemented correctly.
There is no need to attempt to quantify the accuracy by taking the difference between the images obtained from software and hardware simulations as the visual results are very good.
The three image channels processed with both the software and hardware implementation of the demosaicking algorithm are shown for the purposes of visual analysis. The three channels are then recombined together to create the compositie RGB colour image and compared with the colour image obtained from the software simulation as well as the original CFA overlaid image in Figure 3.2.1(iv).
52
(a) (b)
Figure 3.2.1(iii) – Demosaicked images with (a) software simulation and (b) hardware simulation: first row: R channel, second row: G channel and third row: B channel
53
(a) (b)
Figure 3.2.1(iv) – Demosaicked colour image: (a) software simulation (b) hardware simulation
Comparing the images in the Figure 3.2.1(iv) shows the strikingly good result obtained from the hardware simulation in addition to the successful removal of the CFA interference in the demosaicked image. However, on closer inspection, one may observe that there are colour artifacts in regions of high/low frequency discontinuities in the image.
Also, because this image contains a relatively medium amount of high frequency information, one can get away with this linear interpolation demosaicking method. For images with a lot of high frequency information, the limitations of linear methods become ever more apparent. In Figure 3.2.1(v), we present the original CFA overlaid image with the demosaicked results for comparison and the results are even more striking.
The investigation of higher and more advanced methods are left to the reader who wishes to learn more. Some
54
useful sources and research papers are listed at the end of the chapter for further research.
(a) (b) (c)
Figure 3.2.1(v) – (a) Image to be demosaicked (b) Demosaicked image (software) (c) Demosaicked image (hardware simulation)
A snapshot of the ModelSim simulation window is shown in Figure 3.2.1(vi) indicating the clock signal, the inputs and outputs of the interpolation process.
Figure 3.2.1(vi) – Snapshot of VHDL image processing in ModelSim simulation window
The system top level description generated by Xilinx ISE from the VHDL code is shown in Figure 3.2.1(vii). Since we are dealing with unsigned 8-bit images, we only require 8 bits for each channel leading to 256 levels of gray for each
55
channel. The data_out_valid signal and the clock are needed for proper synchronization of the inputs and outputs of the system. Note that this diagram mirrors the black box system description defined at the beginning of this section describing the VHDL implementation of the algorithm.
Figure 3.2.1(vii) – Black box top level VHDL description of demosaicking algorithm
The next level of the top level system shows the major components of the system for each of the R, G and B channels.
Further probing reveals structures similar to those that were described earlier on at the beginning of the VHDL section of this chapter. Refer to the Appendix for more detailed RTL technology schematics and levels of the system.
56
Figure 3.2.1(viii) – first level of VHDL description of demosaicking algorithm
The synthesis results for the implemented demosaicking algorithm on the Xilinx Spartan 3 FPGA chip is given as:
Minimum period: 13.437ns (Maximum Frequency: 74.421MHz) Minimum input arrival time before clock: 6.464ns
Maximum output required time after clock: 10.644ns Maximum combinational path delay: 4.935ns
The maximum frequency implies that for a 256 x 256 image, the frame rate for this architecture is given by:
57
Using this formula yields 1135 frames/sec, which is exceedingly fast.
Using the spatial filter architectures described in Chapter 2, several of the other demosaicking methods can be implemented in VHDL and hardware. Some good papers on image demosaicking are listed in the references section and enable the reader to start implementing the algorithms quickly and performing experiments with the various algorithms.
Summary
In this chapter, the demosaicking process using linear interpolation was described and implemented in software and followed by the VHDL implementation of the linear interpolation algorithm for demosaicking.
References
W. K. Pratt, Digital Image Processing, 4 ed.: Wiley-Interscience, 2007.
Henrique S. etal, “HIGH-QUALITY LINEAR INTERPOLATION FOR DEMOSAICING OF BAYER-PATTERNED COLOR IMAGES”, Microsoft Research, One Microsoft Way, Redmond WA 98052
Alexey Lukin and Denis Kubasov, “An Improved Demosaicing Algorithm”,Faculty of Applied Mathematics and Computer Science, State University of Moscow, Russia
Rémi Jean, “Demosaicing with The Bayer Pattern”, Department of Computer Science, University of North Carolina.
Robert A. Maschal Jr., etal, “Review of Bayer Pattern Color Filter Array (CFA) Demosaicing with New Quality Assessment Algorithms”, ARMY RESEARCH LABORATORY,ARL-TR-5061, January 2010.
Yang-Ki Cho, etal, “Two Stage Demosaicing Algorithm for Color Filter Arrays”, International Journal of Future Generation Communication and Networking, Vol. 3, No. 1, March, 2010.
58
Rajeev Ramanath and Wesley E. Snyder, “Adaptive demosaicking”, Journal of Electronic Imaging 12(4), 633–642 (October 2003).
Boris Ajdin, etal, “Demosaicing by Smoothing along 1D Features”, MPI Informatik, Saarbr¨ucken, Germany.
Yizhen Huang,“Demosaicking Recognition with Applications in Digital Photo Authentication based on a Quadratic Pixel Correlation Model”, Shanghai Video Capture Team, ATI Graphics Division, AMD Inc.
59
Chapter 4
Image Enhancement
This chapter explores some image enhancement concepts, algorithms, their architectures and implementation in VHDL. Image enhancement is a process that involves the improvement of an image by modifying attributes such as contrast, colour, tone and sharpness. This process can be performed manually by a human user or automatically using an image enhancement algorithm, developed as a computer program. Unlike image restoration, image enhancement is a subjective process and usually operates without prior objective image information used to judge the level or quantify the amount of enhancement. Also, enhancement results are usually targeted at human end users who use visual assessment to judge the quality of an enhanced image, which would be difficult for a machine or program to perform.
Image enhancement can be performed in the spatial, frequency, wavelet, and fuzzy domain and in these domains can be classified as local (point and/or mask) or global operations in addition to being linear or nonlinear processes.
A myriad of algorithms have been developed in this field both in industry and in academia evidenced by the numerous conference papers and journals, reports and books and several useful sources are listed at the end of the chapter for further study.
