Image and Video Processing with MATLAB.pdf

(1)

Image and Video Processing Using

MATLAB

April 12, 2011 Rochester, NY

(2)

Agenda

9:00 a.m. Welcome and MathWorks Overview

9:15 a.m. Image Processing with MATLAB 10:30 a.m. Break

10:45 a.m. Video Processing with MATLAB

11:30 a.m. System Design and Hardware Targeting 12:00 p.m. Summary and Wrap-up

(3)

From MathWorks

 Sr. Account Manager – Ed Donovan – [email protected] – 508-647-4251  Application Engineer – Stephanie Kwan

(4)

Agenda

9:00 a.m. Welcome and MathWorks Overview

9:15 a.m. Image Processing with MATLAB

10:30 a.m. Break

(5)

(6)

Common Image Processing Tasks

 Perform segmentation

 Deblur and remove noise

 Measure image features

 Perform edge detection

 Apply morphological operations

 Enhance image contrast

 Perform image registration

(7)

Common Image Processing Challenges

 Reading and writing to various file formats

 Visualizing images and intermediate results

 Testing algorithms with what-if scenarios

 Identifying causes of algorithm failure

 Processing large images with limited memory

(8)

Technical Computing Tasks

Reporting and Documentation

Outputs for Design

Deployment

Share Explore and Discover

Data Analysis and Modeling Algorithm Development Application Development Files Software Hardware Access

(9)

Explore and Discover

MATLAB – A Platform for Technical Computing

Deployment Share Data Analysis and Modeling Algorithm Development Application Development Files Software Hardware Access

(10)

Core MathWorks Products

The leading environment for technical computing

 The industry-standard, high-level programming language

for algorithm development

 Numeric computation

 Data analysis and visualization

 Toolboxes for signal and image processing, statistics, optimization, symbolic math, and other areas

(11)

Image Processing Toolbox

Perform image processing, analysis, visualization, and algorithm development  Image analysis  Image enhancement  Spatial transformation  Image registration  Morphological operations  ROI-based processing

(12)

Demo: Camera Pipeline

From Sensor Data to Image

(13)

1. Noise Reduction 2. Demosaic

3. Tone Mapping 4. White Balance

5. Gamma Correction

(14)

Image Acquisition Toolbox

 _{Acquire images and video directly into}

MATLAB and Simulink

 Configure device properties

 Perform background acquisition

 _{Synchronize multimodal devices}

(15)

Image Acquisition Toolbox Hardware Support

 Industry standard support:

– Frame grabbers

 Analog

 Camera Link

– DCAM compatible FireWire (IIDC 1394)

– GigE Vision

– Common OS webcam interfaces

 Operating system support:

– Windows

– Linux

(16)

Image Acquisition Toolbox Hardware Support

 Manufacturers include:

– Allied Vision Technologies

– Basler – Baumer – DALSA – FLIR – Hamamatsu – Lumenera – Matrox Imaging – National Instruments – PixeLINK – Point Grey – Qimaging – Sony

(17)

Where is Large Imagery Used?

 Satellite imagery

 Aerial surveys

 Super-resolution

 Image sequences and stacks

 Volumetric data

 Multispectral and hyperspectral

(18)

Common Large Image Challenges

 “Out of memory” errors  Slow processing

(19)

Some Steps to Handle Large Images

 Try memory command to see available space

 Preallocate space for variables

 Minimize numerical data types

 Load small portions of an image

 Visualize only what is necessary

 Incorporate more hardware resources

Can lead to

complex code

(20)

blockproc and rsetwrite

 Block processing

– Automatically divides an image into blocks for processing

– Reduces memory usage

– Processes arbitrarily large images

 Reduced resolution data set

– Avoids memory demands in visualizing large images

(21)

(22)

Improving Performance

 MATLAB

– Preallocate space for variables

– Identify bottlenecks with Profiler

– Vectorize code

 Image Processing Toolbox

– Many toolbox functions are faster as of R2010a

 Parallel Computing Toolbox

(23)

Worker Worker Worker Worker Worker Worker Worker Worker

Parallel Computing Toolbox

Going Beyond Serial MATLAB Applications

TOOLBOXES

(24)

Parallel Computing enables you to …

Larger Compute Pool Larger Memory Pool

11 26 41 12 27 42 13 28 43 14 29 44 15 30 45 16 31 46 17 32 47 17 33 48 19 34 49 20 35 50 21 36 51 22 37 52

(25)

Distributing Tasks (Task Parallel)

Proce

(26)

(27)

Parallel Computing Toolbox Support for GPUs

 Automatically use GPU’s with supported built-in

MATLAB functions

– conv2, filter2

 Run supported MATLAB code on the GPU

– arrayfun

 Create kernals from existing CUDA code and PTX files

(28)

Why would you want to use a GPU?

 Speed up execution of computationally intensive simulations

 For example:

(29)

Why use MATLAB for Image Processing?

 Read and write many image file formats

 Visualize and explore images interactively

 Connect directly to cameras and frame grabbers

 Use a large library of built-in functions

 Quickly build custom image processing algorithms

 Block-process large images to avoid memory issues

(30)

Agenda

9:00 a.m. Welcome and MathWorks Overview 9:15 a.m. Image Processing with MATLAB

10:30 a.m. Break

(31)

Agenda

9:00 a.m. Welcome and MathWorks Overview 9:15 a.m. Image Processing with MATLAB 10:30 a.m. Break

10:45 a.m. Video Processing with MATLAB

(32)

Common Video Processing Tasks

 Convert between color spaces

 Enhance video contrast

 Perform segmentation

 De-interlace video frames

 Calculate running statistics

(33)

Common Video Processing Challenges

 Reading and writing to various file formats

 Managing buffers and intermediate storage

 Testing video algorithms with test cases

 Accessing standard video algorithms

 Visualizing results and intermediate steps

(34)

Demo: Working with Video in MATLAB

 Read in and write out data using VideoReader and VideoWriter

 What happens if the video file is very long or the frame size is large?

(35)

System Objects



MATLAB objects that represent algorithms and I/O

capabilities that process data frame by frame



Instantiate and configure

– reader = vision.MultimediaFileReader(‘viptraffic.avi’)



Execute within a loop

– step(reader)



Standard API

– reset – isDone – close

(36)

Video Processing in MATLAB

Need to maintain buffer Explicit indexing Explicit state management

myVid = VideoReader(‘myvideofile.avi’); numFrames = myVid.NumberOfFrames;

numIter = 10;

opticalFlowIn = zeros([size(currentFrame) 5]);

opticalFlowOutput = zeros([size(currentFrame) numFrames]); i = 1; while i <= numFrames opticalFlowIn(:,:,2:end) = opticalFlowIn(:,:,1:end-1); opticalFlowIn(:,:,1) = read(myVid,i); flow = opticalFlow(opticalFlowIn(:,:,1),opticalFlowIn(:,:,5),… ‘horn-schunck’,numIter,‘magitude-squared’); opticalFlowOutput(:,:,i) = flow; i = i+1; end implay(opticalFlowOutput,30)

(37)

Video Processing with System Objects

Initialize objects

“In-the-loop” code is much simpler Implicit states, buffering, and indexing Video player runs in-the-loop

reader = vision.VideoFileReader reader.Filename = ‘myvideofile.avi’; viewer = vision.DeployableVideoPlayer(‘framerate’,30); optical = vision.OpticalFlow optical.Method = ‘horn-schunck’; optical.OutputValue = ‘Magitude-squared’; optical.ReferenceFrameDelay = 3; optical.MaximumIterationCount = 10; while ~isDone(reader) currentFrame = step(reader); OF = step(optical, currentFrame); step(viewer, OF); end

(38)

Demo: Car Detection Using Optical Flow

Use optical flow to detect and count moving

vehicles on a road

(39)

Motion Estimation

 Motion estimation applications:  Object tracking

 Segmentation

 Interpolation

 Compression

 Many methods include:  Optical flow

 Block matching

(40)

Demo: Interpolation with Block Matching

Use block matching to perform motion estimation

and interpolate video

(41)

Computer Vision System Toolbox

Design and simulate computer vision and video processing systems

 Read and write multimedia files

 Display videos

 Overlay text and graphics

 Perform pre- and post-processing

 Perform motion-based processing

 Detect and track objects

 Recognize features

(42)

What is Computer Vision?

 Automating the process of extracting information from images and video to understand a real world scene

Computer Vision Interpretation Detect Identify Classify Recognize Track Pedestrian Bicyclist Truck Car Traffic violation Accident Image Processing Remove noise Adjust contrast Measure …

(43)

Common Computer Vision Applications

 Detect and track objects

 Automatic target recognition

 Count objects in a scene

 Classify or cluster objects

 Automatically register images

 Create mosaics and panoramas

(44)

Statistics Toolbox

Perform statistical analysis, modeling, and algorithm development

 Principle components analysis

 K-means clustering

 Gaussian mixture models

 Naïve Bayes classification

 K-nearest neighbor search

(45)

Why use MATLAB for video processing and

computer vision?

 Read and write to many video file formats

 Manage buffers, indexing, and states with System objects

 Integrate live data from cameras and frame grabbers

 Access algorithms for video processing and computer vision

 Overlay text and graphics annotations on video data

(46)

Agenda

11:30 a.m. System Design and Implementation

(47)

Technical Computing Tasks

Deployment

Share Explore and Discover

Data Analysis and Modeling Algorithm Development Application Development Files Software Hardware Access

(48)

MATLAB Compiler COM _.NET .exe .dll .lib MATLAB Builder EX MATLAB Builder NE MATLAB Builder JA MATLAB Compiler

From MATLAB to Implementation

Simulink Coder MATLAB Coder Simulink HDL Coder Simulink & Stateflow

(49)

MATLAB Compiler

 Automatic conversion of your MATLAB programs into standalone applications and shared libraries

 Support for the full MATLAB language and most toolboxes

(50)

MATLAB Coder

Generate C and C++ Code from MATLAB Code

 Maintain one design in MATLAB

 Design faster and get to C/C++ quickly

 Verify behavior of generated code

 Accelerate computationally intensive portions of MATLAB code

 Test systematically and frequently

 Spend more time improving algorithms in MATLAB

 Compatible with Computer Vision System Toolbox

(51)

 Deploy MATLAB algorithms on Windows/Linux desktop PC

 Integrate MATLAB algorithms with existing C code

 Hand-off code to software engineers for embedded processor implementation

 Accelerate user-written MATLAB algorithms

(52)

Broad support for Features and Functions

 Broad set of language features and functions/system objects supported for code generation.

Matrices and

Arrays Data Types

Programming Constructs Functions • Matrix operations • N-dimensional arrays • Subscripting • Frames • Persistent variables • Global variables • Complex numbers • Integer math • Double/single-precision • Fixed-point arithmetic • Characters • Structures • Numeric classes • Variable-sized data • System objects • Arithmetic, relational, and logical operators • Program control

(if, for, while, switch )

• MATLAB functions and

sub-functions

• Variable length argument lists • Function handles

Supported algorithms

• > 400 MATLAB operators and

functions

• > 200 System objects for • Signal processing • Communications • Computer vision

(53)

Demo: Camera Pipeline Revisited

(54)

Targeting DSPs and FPGAs

MATLAB and Simulink

• Algorithm development • Debugging and profiling • System design

Generate code _{Verify design}

Fixed-Point Modeling

Link to Embedded Software

(55)

Demo: Camera Pipeline System Design

From Sensor Data to Image

(56)

Why use MATLAB and Simulink for embedded

hardware design?

 Automatically convert MATLAB to C code

 Reduce the number of versions of code

 Simplify the design process

 Reuse MATLAB and legacy code in models

 Convert floating-point algorithms to fixed point

 Optimize algorithms for DSPs and FPGAs

(57)

Agenda

11:30 a.m. System Design and Hardware Targeting

(58)

What You’ve Seen Today

MATLAB for image and video processing

 Image Processing Toolbox

– Visualize, analyze, and process images

 Image Acquisition Toolbox

– Test image processing algorithms with live data

 Computer Vision System Toolbox

– Visualize, analyze, and process video

– Motion estimation, tracking, and stereo vision

 Parallel Computing Toolbox

(59)

Doheny Eye Institute Develops

Next-Generation of Retinal Prosthesis with MathWorks Tools

Challenge

Develop next-generation, higher-resolution retinal prostheses

Solution

Use MathWorks tools to develop, simulate, and automatically generate code for real-time image processing algorithms

Results

 Development time reduced from months to weeks  DSP deployment streamlined

 Patient testing improved

“With Video and Image Processing Blockset and Target Support

Package, we rapidly prototype our image and video processing

algorithms on the DM642 board. This can save me days or weeks of time.”

Neha Parikh Doheny Eye Institute Illustration of a retinal prosthesis prototype.

(60)

Given Imaging Develops Camera-in-a-Capsule Using MATLAB to Improve the Diagnosis of Gastrointestinal Disorders

Challenge

Create an alternative to endoscopy and other invasive gastrointestinal imaging procedures

Solution

Use MATLAB and companion toolboxes to develop and implement a swallowable video capsule

Results

 Fast, efficient development

 Easy access to precise diagnostic information  Improved patient care

“With MATLAB, we simulated the intended system and fine-tuned it at the early stages of implementation, enabling us to develop critical engineering programs that met requirements on the first iteration.”

(61)

Additional Articles and User Stories

(62)

From a Leading Textbook Author ...

"I have used a number of commercial image processing packages over the years, and prefer the MathWorks

Image Processing Toolbox for several reasons: the wide variety of functions it provides, the user’s ability to write

additional functions with minimal effort, the quality of the software, and the high level of support."

(63)

(64)

Migration Planning Component Deployment Full Application Deployment Cont inuo us Im prov ement

Consulting Services

Accelerating return on investment

A global team of experts supporting every stage of tool and process integration

Advisory Services

Process Assessment

Jumpstart

Process and Technology Standardization

Process and Technology Automation

(65)

Training Services

Exploit the full potential of MathWorks products

Flexible delivery options:

 Public training available worldwide

 Onsite training with standard or customized courses

 Web-based training with live, interactive instructor-led courses

 Self-paced interactive online training

More than 30 course offerings:

 Introductory and intermediate training on MATLAB, Simulink, Stateflow, code generation, and Polyspace products

 Specialized courses in control design, signal processing, parallel computing, code generation, communications, financial analysis,

(66)

MATLAB Central

 Community for MATLAB and Simulink users

 Over 1 million visits per month

 File Exchange

– Upload/download access to free files

including MATLAB code, Simulink models, and documents

– Ability to rate files, comment, and ask questions

– More than 12,500 contributed files, 300 submissions per month, 50,000 downloads per month

 Newsgroup

– Web forum for technical discussions about MathWorks products

– More than 300 posts per day

 Blogs

(67)

Connections Program

More than 400 add-on products and services that complement and extend MathWorks products:

 Specialized third-party toolboxes for MATLAB

 Interfaces to third-party software and hardware products

 Specialized training courses and consulting services

 System integrators and suppliers that incorporate MathWorks products

(68)

Book Program

More than 1200 books for educational and professional use, in 26 languages  Mathematics  Aerospace engineering  Environmental sciences  Chemistry  Finance  Electronics  Controls  Signal processing  Image processing  Biosciences  Communications  Mechanical engineering

(69)

Technical Support

Resources

 Over 100 support engineers

– All with MS degrees (EE, ME, CS)

– Local support in North America, Europe, and Asia

 Comprehensive, product-specific Web support resources

High customer satisfaction

 95% of calls answered within three minutes

 70% of issues resolved within 24 hours  80% of customers surveyed rate satisfaction at 80–100%     

(70)

For More Information

 Experiment with product by downloading a trial

 Peruse videos, webinars, user stories, and demos online

 Contact us

– Talk to a sales representative to get answers to your questions

– Discuss your projects

with MathWorks applications engineers