Lecture 9 Image formation

Full text

(1)Antonio Torralba, 2012. Lecture 9 Image formation.

(2) Image formation 3D world. 2D image. Point of observation. Figures © Stephen E. Palmer, 2002.

(3) Cameras, lenses, and calibration • Camera models • Projection equations. Images are projections of the 3-D world onto a 2-D plane….

(4) The structure of ambient light .

(5) The structure of ambient light .

(6) The Plenop5c Func5on Adelson & Bergen, 91. The intensity P can be parameterized as:. P (θ, φ, λ, t, X, Y, Z) “The complete set of all convergence points constitutes the permanent possibilities of vision.” Gibson.

(7) Measuring the Plenop5c func5on . Why is there no picture appearing on the paper?.

(8) Light rays from many different parts of the scene strike the same point on the paper.. Forsyth & Ponce.

(9) Measuring the Plenop5c func5on The camera obscura The pinhole camera.

(10) Light rays from many different parts of the scene strike the same point on the paper.. Forsyth & Ponce. The pinhole camera only allows rays from one point in the scene to strike each point of the paper..

(11) Pinhole camera . Photograph by Abelardo Morell, 1991.




(15) Problem Set 1 . http://www.foundphotography.com/PhotoThoughts/archives/2005/04/pinhole_camera_2.html.

(16) Problem Set 1 .

(17) Effect of pinhole size . Wandell, Foundations of Vision, Sinauer, 1995.

(18) Wandell, Foundations of Vision, Sinauer, 1995.

(19) Animal Eyes . Animal Eyes. Land & Nilsson. Oxford Univ. Press.

(20) Measuring distance . • Object size decreases with distance to the pinhole • There, given a single projection, if we know the size of the object we can know how far it is. • But for objects of unknown size, the 3D information seems to be lost..

(21) Playing with pinholes .

(22) Two pinholes .

(23) Two pinholes . What is the minimal distance between the two projected images?.

(24) Anaglyph pinhole camera .



(27) Synthesis of new views. Anaglyph.

(28) Problem set • Build the device • Take some pictures and put them in the report • Take anaglyph images • Work out the geometry • Recover depth for some points in the image .

(29) Shadows?.

(30)

(31)

(32) Accidental pinhole camera .

(33)

(34)

(35) Window turned into a pinhole. View outside.

(36) Source: wikipedia. ”a camera obscura has been used ... to bring images from the outside into a darkened room”. Chris Fraser. Aberlado Morell.

(37)

(38)

(39) Window open. Window turned into a pinhole.

(40)

(41) Making a pinhole with home materials .

(42) Making a pinhole with home materials .

(43) An hotel room, contrast enhanced.. The view from my window. Accidental pinholes produce images that are unnoticed or misinterpreted as shadows.

(44) Another hotel room.

(45)

(46)

(47)

(48)

(49)

(50)

(51) Accidental pinholes in outdoor scenes . Pierre Moreels father (source: facebook).

(52) Accidental pinhole camera . * Aperture Outside scene See Zomet, A.; Nayar, S.K. CVPR 2006 for a detailed analysis..

(53) Visualizing the convolution.

(54) An5-‐pinhole or Pinspeck cameras Adam L. Cohen, 1982.

(55) Pinhole and An5-‐pinhole cameras pinhole. Anti-pinhole. …. Intensity. …. Intensity. location. location. Adam L. Cohen, 1982.

(56) Natural eyes Lenses. Pinholes. Anti-pinholes. nautilus. Euglena?.

(57) Mixed accidental pinhole and an5-‐pinhole cameras .

(58) Mixed accidental pinhole and an5-‐pinhole cameras .

(59) Mixed accidental pinhole and an5-‐pinhole cameras Room with a window. Person in front of the window. Difference image. -. =. -. = ?.

(60) Mixed accidental pinhole and anti-pinhole cameras .

(61) Mixed accidental pinhole and anti-pinhole cameras Body as the occluder. View outside the window.

(62) Looking for a small accidental occluder t1 .. .. . t2 .. .. . t3.

(63) Reference. Video -. =.

(64) Looking for a small accidental occluder Body as the occluder. Hand as the occluder. View outside the window.

(65) Camera Models .

(66) Right - handed system y. y. left top far. z x x. z. right bottom near. y. z x z. x y.

(67) Perspective projection camera. world f z. y. y’. Cartesian coordinates: We have, by similar triangles, that (x, y, z) -> (f x/z, f y/z, -f) Ignore the third coordinate, and get. (x, y,z) → ( f. €. x y ,f ) z z.

(68) Geometric properties of projection • • • • . Points go to points Lines go to lines Planes go to whole image or half-planes. Polygons go to polygons. • Degenerate cases – line through focal point to point – plane through focal point to line.

(69) Vanishing point. y z. camera.

(70) http://www.ider.herts.ac.uk/school/courseware/ graphics/two_point_perspective.html.

(71) Vanishing points • Each set of parallel lines (=direction) meets at a different point – The vanishing point for this direction. • Sets of parallel lines on the same plane lead to collinear vanishing points. – The line is called the horizon for that plane.

(72) Perspective projection of that line. Line in 3-space. x(t ) = x0 + at y (t ) = y0 + bt z (t ) = z0 + ct. In the limit as we have (for c. fx f (x 0 + at) x'(t) = = z z0 + ct fy f (y 0 + bt) y'(t) = = z z0 + ct. t → ±∞ ≠ 0 ):€. This tells us that any set of parallel lines (same a, b, c parameters) project to the same point (called the vanishing point).. fa x'(t) " "→ c fb y'(t) " "→ c.

(73) What if you photograph a brick wall head-on?. y. x.

(74) Brick wall line in 3-space. x(t ) = x0 + at y (t ) = y0 z (t ) = z0. Perspective projection of that line. f ⋅ (x 0 + at) x'(t) = z0 f ⋅ y0 y'(t) = z0. All bricks have same z0. Those in same row have same y0 Thus, a brick wall, photographed head-on, gets rendered as set of parallel lines in the image plane.. €.

(75) Other projection models: Orthographic projection. ( x, y , z ) → ( x , y ).

(76) Other projection models: Weak perspective • Issue – perspective effects, but not over the scale of individual objects – collect points into a group at about the same depth, then divide each point by the depth of its group – Adv: easy – Disadv: only approximate. ⎛ fx fy ⎞ ( x, y, z ) → ⎜⎜ , ⎟⎟ ⎝ z0 z0 ⎠.

(77) Three camera projections 3-d point. (1) Perspective: (2) Weak perspective: (3) Orthographic:. 2-d image position. ⎛ fx fy ⎞ ( x, y, z ) → ⎜ , ⎟ ⎝ z z ⎠ ⎛ fx fy ⎞ ( x, y, z ) → ⎜⎜ , ⎟⎟ ⎝ z0 z0 ⎠ ( x, y , z ) → ( x, y ).

(78) Three camera projections. Perspective projection. Parallel (orthographic) projection. Weak perspective?.

(79) Homogeneous coordinates • Is this a linear transformation? • no—division by z is nonlinear. Trick: add one more coordinate:. homogeneous image coordinates. homogeneous scene coordinates. Converting from homogeneous coordinates. Slide by Steve Seitz.

(80) Perspective Projection • Projection is a matrix multiply using homogeneous coordinates:. "1 0 0 $ $0 1 0 $#0 0 1/ f. " x% " x % 0%$ ' '$ y' = $ y ' 0' $ ' $ z' $#z / f '& 0'&$ ' #1&. # x y& ⇒%f , f ( $ z z'. €. €. This is known as perspective projection • The matrix is€ the projection matrix. Slide by Steve Seitz.

(81) €. Perspective Projection How does scaling the projection matrix change the transformation?. "1 0 0 $ $0 1 0 $#0 0 1/ f "f $ $0 $# 0. 0 f 0. 0 € 0 1. " x% " x % 0%$ ' '$ y' = $ y ' 0' $ ' $ z' $#z / f '& 0'&$ ' #1&. # x y& ⇒%f , f ( $ z z'. " x% 0%$ ' '$ y' 0' $ z' 0'&$ ' #1&. # x y& ⇒%f , f ( $ z z'. € " fx%. $ ' = $ fy' $# z '& €. €. Slide by Steve Seitz.

(82) Orthographic Projection Special case of perspective projection • Distance from the COP to the PP is infinite Image. World. • Also called “parallel projection” • What’s the projection matrix?. ? Slide by Steve Seitz.

(83) Orthographic Projection Special case of perspective projection • Distance from the COP to the PP is infinite Image. World. • Also called “parallel projection” • What’s the projection matrix?. Slide by Steve Seitz.

(84) Homogeneous coordinates 2D Points: " x% p =$ ' # y&. 2D Lines:. " x'% $ ' p'= $ y'' $#w''&. " x% $ ' p'= $ y' $#1'&. ax + by + c = 0. €. " x% $ ' a b c [ € ]$y' = 0 $#1'&. €. "x' /w'% p =$ ' y' /w' # &. €. l = [a b c ] ⇒ [n x d. €. ny (nx, ny). d].

(85) Homogeneous coordinates Intersection between two lines: x12. a2 x + b2 y + c 2 = 0 €. €. a1 x + b1 y + c1 = 0. l1 = [ a1 b1 c1 ] x12 = l1 × l2. l2 = [ a2 b2 c 2 ] € €.

(86) Homogeneous coordinates Line joining two points: p1 p2 €. ax + by + c = 0. € € p1 = [ x1. y1 1] l = p1 × p2. p2 = [ x 2. y 2 1]. € €.

(87) 2D Transformations.

(88) 2D Transformations. Example: translation. =. +. tx ty.

(89) 2D Transformations. Example: translation. =. +. tx ty. =. 1. 0. tx. 0. 1. ty. . 1.

(90) 2D Transformations. Example: translation. =. +. tx ty. =. 1. 0. tx. 0. 1. ty. .. = 1. 1. 0. tx. 0. 1. ty. 0. 0. 1. .. Now we can chain transformations.

(91) Translation and rotation, written in each set of coordinates.    B B A B p= A R p+ A t. Non-homogeneous coordinates.  B A p= A C p. Homogeneous coordinates. B. € €. where. #− − − % B − R − B A % C = A %− − − % $ 0 0 0. | &  B ( At( | ( ( 1'.

(92) €. Translation and rotation “as described in the coordinates of frame B” Let’s write. iÂ.  B  B A B p= A R p+ A t. A. €. py. ˆj A. as a single matrix equation:. " B px % " − − − $B ' $ B p $ y ' = $− A R − $ B pz ' $ − − − $ ' $ 0 0 0 # 1 & #. A. €. | B A. t. | 1. €€. A " % px % '$ A p€ ' '$ y ' A $ ' pz ' '$ ' &# 1 &.  p. px A. pz kÂ B A.  t.

(93) Camera calibration Use the camera to tell you things about the world: – Relationship between coordinates in the world and coordinates in the image: geometric camera calibration, see Szeliski, section 5.2, 5.3 for references – (Relationship between intensities in the world and intensities in the image: photometric image formation, see Szeliski, sect. 2.2.).

(94) Intrinsic parameters: from idealized world coordinates to pixel values. Forsyth&Ponce. Perspective projection. x u= f z y v= f z.

(95) Intrinsic parameters. But “pixels” are in some arbitrary spatial units. x u =α z y v =α z.

(96) Intrinsic parameters. Maybe pixels are not square. x u =α z y v=β z.

(97) Intrinsic parameters. We don’t know the origin of our camera pixel coordinates. x u = α + u0 z y v = β + v0 z.

(98) Intrinsic parameters. v vʹ′ θ. u ʹ′. u vʹ′ sin(θ ) = v uʹ′ = u − cos(θ )vʹ′ = u − cot(θ )v May be skew between camera pixel axes. x y u = α − α cot(θ ) + u0 z z β y v= + v0 sin(θ ) z.

(99) Intrinsic parameters, homogeneous coordinates. x y − α cot(θ ) + u0 z z β y v= + v0 sin(θ ) z u =α. Using homogenous coordinates, we can write this as:. " x% " % " u% α −α cot(θ ) u0 0 $ ' ' y β $ ' $ $ ' v = 0 v 0 $ ' 0 $ ' $ z' sin(θ ) $ ' $ ' # 1& # 0 0 1 0&$ 1 ' # &. or:. In pixels.  p. =. K. C. In camera-based coords.  p.

(100) Extrinsic parameters: translation and rotation of camera frame C.  C W  C p=W R p+ W t. " % "− − − $ C ' $ C p − − $ '=$ WR $ ' $− − − $ ' $ 0 0 0 # & #. Non-homogeneous coordinates. | %" %  $ ' C ' W p W t '$ ' | '$ ' ' 1 &$# '&. Homogeneous coordinates.

(101) Combining extrinsic and intrinsic calibration parameters, in homogeneous coordinates  C p = K p. pixels. Camera coordinates. €. € Forsyth&Ponce. " % "− − − $ C ' $ C p − − $ '=$ WR $ ' $− − − $ ' $ 0 0 0 # & #.  C p = K(W R. 000.  p = M €. W.  p. Intrinsic. World coordinate. | %" %  $ ' C ' W p W t '$ ' | '$ ' '$ ' 1 &# & C W.  t). 1. W.  p. Extrinsic.

(102) Other ways to write the same equation pixel coordinates.  p= M. W. " u% ". m1T $ ' $ T $ v ' = $. m2 $ ' $ T 1 . m # & # 3.  p. world coordinates. " W px % . .%$ W ' '$ py ' . .' W $ pz ' ' . .&$ ' # 1 &. Conversion back from homogeneous coordinates leads to:.  m1 ⋅ P  u= m3 ⋅ P m2 ⋅ P  v= m3 ⋅ P.

(103) Camera parameters A camera is described by several parameters • • • • . Translation T of the optical center from the origin of world coords Rotation R of the image plane focal length f, principle point (x’c, y’c), pixel size (sx, sy) blue parameters are called “extrinsics,” red are “intrinsics”. Projection equation ⎡ sx⎤ ⎡* * * *⎤ x = ⎢⎢sy⎥⎥ = ⎢⎢* * * *⎥⎥ ⎢⎣ s ⎥⎦ ⎢⎣* * * *⎥⎦. ⎡ X ⎤ ⎢ Y ⎥ ⎢ ⎥ = ΠX ⎢ Z ⎥ ⎢ ⎥ ⎣ 1 ⎦. • The projection matrix models the cumulative effect of all parameters • Useful to decompose into a series of operations identity matrix. $− fsx & Π =& 0 &% 0. 0 − fsy 0. intrinsics. x'c '$1 0 0 0'$ )& )R y'c )&0 1 0 0)& 3x 3 &0 1 ()%&0 0 1 0)(% 1x 3 projection. 0 3x1'$I 3x 3 )& 1 )(&% 01x 3. rotation. ' ) 1 )(. T. 3x1. translation. • The definitions of these parameters are not completely standardized €. – especially intrinsics—varies from one book to another.

(104)

No results found