Matting and Compositing

(1)

Matting and Compositing

Digital Visual Effects Yung-Yu Chuang

(2)

Outline

• Traditional matting and compositing

• The matting problem

• Bayesian matting and extensions

• Matting with less user inputs

• Matting with multiple observations

• Beyond the compositing equation*

• Conclusions

(3)

Outline

• Conclusions

(4)

Photomontage

The Two Ways of Life, 1857, Oscar Gustav Rejlander Printed from the original 32 wet collodion negatives.

(5)

Photographic compositions

Lang Ching-shan

(6)

Use of mattes for compositing

The Great Train Robbery (1903) matte shot

(7)

Use of mattes for compositing

The Great Train Robbery (1903) matte shot

(8)

Optical compositing

King Kong (1933) Stop-motion + optical compositing

(9)

Digital matting and compositing

The lost world (1925) The lost world (1997)

Miniature, stop-motion Computer-generated images

(10)

Digital matting and composting

King Kong (1933) Jurassic Park III (2001)

Optical compositing

Blue-screen matting, digital composition, digital matte painting

(11)

Oscar award, 1996

Smith Duff Catmull Porter

(12)

Titanic

Matting and Compositing

(13)

Matting and Compositing

background replacement

background editing

(14)

Digital matting: bluescreen matting

Forrest Gump (1994)

• The most common approach for films.

• Expensive, studio setup.

• Not a simple one-step process.

(15)

Color difference method (Ultimatte)

Blue-screen photograph

C=F+B

Spill suppression if B>G then B=G

F

Matte creation

=B-max(G,R) 

demo with Paint Shop Pro (B=min(B,G))

(16)

Problems with color difference

Background color is usually not perfect! (lighting, shadowing…)

(17)

Chroma-keying (Primatte)

(18)

Chroma-keying (Primatte)

demo

(19)

Outline

• Conclusions

(20)

Compositing

B F

C  α  (1 α)

F  B

C

foreground color alpha matte background plate composite

compositing equation

B

F C

=0

(21)

Compositing

B F

C  α  (1 α)

F  B

C

composite

B

F

C =1

(22)

Compositing

B F

C  α  (1 α)

F  B

C

composite

B

C F

=0.6

(23)

Matting

C

observation

B F

C  α  (1 α)

F  B

(24)

Matting

C C  αF  (1 α)B

F  B

Three approaches:

1 reduce #unknowns 2 add observations 3 add priors

(25)

Matting (reduce #unknowns)

C

F BB

B F

C  α  (1 α)

difference matting



(26)

Matting (reduce #unknowns)

C F

B F

C  α  (1 α) B

blue screen matting



(27)

Matting (add observations)

F

B F

C  α  (1 α)

triangulation



B F

C  α  (1 α) B

C

(28)

Natural image matting

B C

Matting (add priors)

F

B F

C  α  (1 α)

 B

rotoscoping Ruzon-Tomasi

FG BG

unknown

(29)

Outline

• Conclusions

(30)

Bayesian framework

f(z)+

z y

para- meters

observed signal )

| ( max

* P z y

z  z

) (

) ( )

| max (

y P

z P z

y P

 z

) ( )

| (

max L y z L z

z 



Example:

super-resolution de-blurring

de-blocking

…

(31)

Bayesian framework

) ( )

| (

max

* L y z L z

z  z 

2

) 2

(



z f

data y 

evidence

a-priori knowledge

f(z)+

z y

para- meters

observed signal

(32)

Bayesian framework

likelihood priors posterior probability

(33)

Priors

(34)

Bayesian matting

(35)

Optimization

repeat

until converge 1. fix alpha

2. fix F and B

(36)

(37)

(38)

(39)

(40)

(41)

(42)

(43)

(44)

(45)

(46)

(47)

(48)

(49)

(50)

(51)

(52)

(53)

Demo

(54)

input trimapalpha

Results

(55)

input composite

Results

(56)

Comparisons

input imagetrimap

(57)

Comparisons

Bayesian Ruzon-Tomasi

(58)

Comparisons

Bayesian Ruzon-Tomasi

(59)

Comparisons

Mishima

(60)

Comparisons

Bayesian

(61)

Comparisons

input image

(62)

Comparisons

Bayesian Mishima

(63)

Comparisons

Bayesian Mishima

(64)

input video

Video matting

(65)

input key

trimaps input video

Video matting

(66)

interpo- lated trimaps

input video

Video matting

(67)

input video

output alpha

Video matting

(68)

Compo- site

input video

output alpha

Video matting

(69)

(70)

optical flow

(71)

optical flow

(72)

(73)

(74)

(75)

(76)

(77)

(78)

(79)

(80)

Sample composite

(81)

Garbage mattes

(82)

Garbage mattes

(83)

Background estimation

(84)

Background estimation

(85)

Alpha matte

(86)

Comparison

without background

with

background

(87)

(88)

(89)

(90)

(91)

(92)

(93)

C

B P(F)

(94)

(95)

(96)

(97)

Problems with Bayesian matting

• It requires fine trimaps for good results

• It is tedious to generate fine trimaps

• Its performance rapidly degrades when foreground and background patterns

become complex

• There is no direct and local control to the resulting mattes

(98)

Outline

• Conclusions

(99)

Scribble-based input

trimap scribble

(100)

Motivation

(101)

LazySnapping

(102)

LazySnapping

n-th mean foreground color

(103)

LazySnapping

(104)

LazySnapping

(105)

Matting approaches

• Sampling approaches: solve for each alpha separately by utilizing local

fg/bg samples, e.g. Ruzon/Tomasi, Knockout and Bayesian matting.

• Propagation approaches: solve the

whole matte together by optimizing, e.g. Poisson, BP, random walker,

closed-form and robust matting.

(106)

Poisson matting

(107)

Poisson matting

(108)

Robust matting

• Jue Wang and Michael Cohen, CVPR 2007

(109)

Robust matting

• Instead of fitting models, a non- parametric approach is used

Bayesian Robust

(110)

Robust matting

• We must evaluate hypothesized foreground/background pairs

B^j

Fⁱ C

distance ratio

(111)

Robust matting

• To encourage pure fg/bg pixels, add weights

B F¹

C F²

(112)

Robust matting

• Combine them together. Pick up the best 3 pairs and average them

confidence

(113)

Robust matting

(114)

Robust matting

matte confidence

(115)

Matte optimization

Solved by Random Walk Algorithm

(116)

Matte optimization

data constraints

neighborhood constraints

(117)

Demo (EZ Mask)

(118)

Evaluation

• 8 images collected in 3 different ways

• Each has a “ground truth” matte

(119)

(120)

Evaluation

• Mean square error is used as the accuracy metric

• Try 8 trimaps with different accuracy for testing robustness

• 7 methods are tested: Bayesian,

Belief propagation, Poisson, Random Walk, KnockOut2, Closed-Form and Robust matting

(121)

Quantitative evaluation

(122)

Subjective evaluation

(123)

Subjective evaluation

(124)

Ranks of these algorithms

Poisson

Random walk Knockout2

Bayesian

Belief Propagation Close-form

Robust matting

accuracy 6.9

6.0 4.5 3.9 3.3 2.6 1.0

robustness 6.8

4.4 4.5 6.0 3.1 2.0 1.3

(125)

Summary

• Propagation-based methods are more robust

• Sampling-based methods often

generate more accurate mattes than propagation-based ones with fine

trimaps

• Robust matting combines strengths of both

(126)

New evaluation (CVPR 2009)

• http://www.alphamatting.com/

(127)

Soft scissor

• Jue Wang et. al., SIGGRAPH 2007

• Users interact in a similar way to intelligent scissors

(128)

Flowchart

(129)

Flowchart

(130)

Soft scissor

(131)

Demo (Power Mask)

(132)

Outline

• Conclusions

(133)

Matting with multiple observations

• Invisible lights

– Polarized lights – Infrared

• Thermo-key

• Depth Keying (ZCam)

• Flash matting

(134)

Invisible lights (Infared)

(135)

Invisible lights (Infared)

(136)

Invisible lights (Infared)

(137)

Invisible lights (Infared)

(138)

Invisible lights (Infared)

(139)

Invisible lights (Infared)

(140)

Invisible lights (Polarized)

(141)

Invisible lights (Polarized)

(142)

Thermo-Key

(143)

Thermo-Key

(144)

ZCam

(145)

Defocus matting

(146)

video

(147)

Matting with camera arrays

video

(148)

Flash matting

flash no flash matte

(149)

Flash matting

Background is much further than foreground and receives almost no flash light

(150)

Flash matting

Foreground flash matting equation

Generate a trimap and directly apply Bayesian matting.

(151)

Foreground flash matting

(152)

Joint Bayesian flash matting

(153)

Joint Bayesian flash matting

(154)

Comparison

flash no flash

(155)

Comparison

foreground flash matting

ioint Bayesian flash matting

(156)

Flash matting

(157)

Outline

• Conclusions

(158)

Conclusions

• Matting algorithms improves a lot in these 10 years

• In production, it is still always

preferable to shoot against uniform backgrounds

• Algorithms for more complex backgrounds

• Devices or algorithms for automatic matting

(159)

Thanks for your attention!

(160)

Shadow matting

and composting

(161)

target background source scene

(162)

target background blue screen image

(163)

target background blue screen composite

(164)

photograph blue screen composite

(165)

Geometric errors

(166)

Photometric errors

(167)

(168)

(169)

S L

C



(170)

S L

C



shadow

Shadow compositing equation

C=L+(1-)S

(171)

C



shadow

Shadow matting

C=L+(1-)S

(172)

S L

C



shadow

Shadow matting

C=L+(1-)S

(173)

S L

C



shadow

Shadow matting



C=L+(1-)S

(174)

C

S  L

shadow

Shadow compositing

C=L+(1-)S

(175)

(176)

(177)

(178)

(179)

(180)

(181)

Geometric errors

(182)

(183)

Requirement #1

(184)

Requirement #2

(185)

(186)

(187)

(188)

(189)

(190)

(191)

(192)

(193)

(194)

(195)

(196)

(197)

(198)

(199)

(200)