SlideShare a Scribd company logo

transformations2 and fitting on image processing.pptx

Download as PPTX, PDF
••
Indra Hermawan, profile picture
Indra Hermawan

transformations2 and fitting on image processing.pptx

Devices & Hardware
1 of 80
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
Transformation s and Fitting EECS 442 – David Fouhey Winter 2023, University of Michigan https://web.eecs.umich.edu/~fouhey/teaching/EECS442_W23/
Administrivia • Discussion this week = office hours
So Far 1. How do we find distinctive / easy to locate features? (Harris/Laplacian of Gaussian) 2. How do we describe the regions around them? (histogram of gradients) 3. How do we match features? (L2 distance) 4. How do we handle outliers? (RANSAC)
Today As promised: warping one image to another
Why Mosaic? • Compact Camera FOV = 50 x 35° Slide credit: Brown & Lowe
Why Mosaic? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135° Slide credit: Brown & Lowe
Why Mosaic? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135° • Panoramic Mosaic = 360 x 180° Slide credit: Brown & Lowe
Why Bother With This Math? Slide credit: A. Efros
Homework 1 Style Translation only via alignment Slide credit: A. Efros
Result Slide credit: A. Efros
Image Transformations f x T g x f x T g x Image filtering: change range of image 𝑔( 𝑥)=𝑇 ( 𝑓 ( 𝑥)) 𝑔 ( 𝑥 )= 𝑓 ¿ Image warping: change domain of image Slide credit: A. Efros
Image Transformations T T Image filtering: change range of image 𝑔(𝑥 , 𝑦)=𝑇 ( 𝑓 (𝑥, 𝑦 )) 𝑔 ( 𝑥 , 𝑦 )= 𝑓 ¿ Image warping: change domain of image f g f g Slide credit: A. Efros
Parametric (Global) warping translation rotation aspect affine perspective cylindrical Examples of parametric warps Slide credit: A. Efros
Parametric (Global) Warping T p’ = (x’,y’) T is a coordinate changing machine p = (x,y) Note: T is the same for all points, has relatively few parameters, and does not depend on image content 𝒑′ =𝑇 (𝒑) Slide credit: A. Efros
Parametric (Global) Warping T p’ = (x’,y’) p = (x,y) Today we’ll deal with linear warps 𝒑′ ≡𝑻𝒑 T: matrix; p, p’: 2D points. Start with normal points and =, then do homogeneous cords and ≡ Slide credit: A. Efros
Scaling  2 Scaling multiplies each component (x,y) by a scalar. Uniform scaling is the same for all components. Note the corner goes from (1,1) to (2,2) Slide credit: A. Efros
Scaling Non-uniform scaling multiplies each component by a different scalar. X  2, Y  0.5 Slide credit: A. Efros
Scaling What does T look like? 𝑥′ =𝑎𝑥 𝑦 ′ =𝑏𝑦 Let’s convert to a matrix: [𝑥 ′ 𝑦 ′ ]= [𝑎 0 0 𝑏][𝑥 𝑦 ] scaling matrix S What’s the inverse of S? Slide credit: A. Efros
2D Rotation Rotation Matrix But wait! Aren’t sin/cos non-linear? x’ is a linear combination/function of x, y x’ is not a linear function of θ What’s the inverse of Rθ? 𝑰 =𝑹𝜽 𝑇 𝑹𝜽 [𝑥 ′ 𝑦 ′ ]= [c os ⁡(𝜃) − sin (𝜃 ) sin ( 𝜃) cos(𝜃 ) ][𝑥 𝑦 ] Slide credit: A. Efros
Things You Can Do With 2x2 Identity / No Transformation Shear [𝑥 ′ 𝑦 ′ ]= [ 1 𝑠 h𝑥 𝑠h𝑦 1 ][𝑥 𝑦 ] [𝑥 ′ 𝑦 ′ ]= [1 0 0 1 ][𝑥 𝑦 ] Slide credit: A. Efros
Things You Can Do With 2x2 2D Mirror About Y-Axis [𝑥 ′ 𝑦 ′ ]= [−1 0 0 1 ][𝑥 𝑦 ] Before After 2D Mirror About X,Y [𝑥 ′ 𝑦 ′ ]= [−1 0 0 −1][𝑥 𝑦 ] Before After Slide credit: A. Efros
What’s Preserved? Projections of parallel 3D lines are not necessarily parallel, so not parallelism 3D lines project to 2D lines so lines are preserved Distant objects are smaller so size is not preserved
What’s Preserved With a 2x2 [𝑥 ′ 𝑦 ′ ]= [𝑎 𝑏 𝑐 𝑑][𝑥 𝑦 ]=𝑇 [𝑥 𝑦] After multiplication by T (irrespective of T) • Origin is origin: 0 = T0 • Lines are lines • Parallel lines are parallel
Things You Can’t Do With 2x2 What about translation? x’ = x + tx, y’ = y+ty +(2,2) How do we make it linear?
Homogeneous Coordinates Again What about translation? x’ = x + tx, y’ = y+ty +(2,2) [ 𝑥+𝑡𝑥 𝑦 +𝑡 𝑦 1 ]≡ [ 𝑥′ 𝑦 ′ 1 ]≡ [ 1 0 𝑡𝑥 0 1 𝑡𝑦 0 0 1 ][ 𝑥 𝑦 1 ] Slide credit: A. Efros
Representing 2D Transformations How do we represent a 2D transformation? Let’s pick scaling [ 𝑥′ 𝑦 ′ 1 ]≡ [ 𝑠𝑥 0 𝑎 0 𝑠𝑦 𝑏 𝑑 𝑒 𝑓 ][ 𝑥 𝑦 1 ] a b d e f 0 0 0 0 1 What’s
Affine Transformations Affine: linear transformation plus translation In general (without homogeneous coordinates) 𝒙 ′= 𝑨𝒙+𝒃 Will the last coordinate w’ always be 1? [ 𝑥 ′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 0 0 1 ][ 𝑥 𝑦 1 ] t
Matrix Composition [ 𝑥′ 𝑦 ′ 𝑤 ′]≡ [ 1 0 𝑡𝑥 0 1 𝑡 𝑦 0 0 1 ][ cos (𝜃 ) −sin ( 𝜃) 0 sin (𝜃) cos ( 𝜃) 0 0 0 1][ 𝑠𝑥 0 0 0 𝑠 𝑦 0 0 0 1 ][ 𝑥 𝑦 𝑤] 𝑇 (𝑡𝑥 ,𝑡𝑦 ) 𝑅( 𝜃) 𝑆(𝑠𝑥 ,𝑠𝑦 ) We can combine transformations via matrix multiplication. Does order matter? Slide credit: A. Efros
What’s Preserved With Affine After multiplication by T (irrespective of T) • Origin is origin: 0 = T0 • Lines are lines • Parallel lines are parallel [ 𝑥′ 𝑦 ′ 1 ]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 0 0 1 ][ 𝑥 𝑦 1 ]≡ 𝑻 [ 𝑥 𝑦 1 ]
Homogeneous Equivalence z x y [x,y,w] λ[x,y,w] Two homogeneous coordinates are equivalent if they are proportional to each other. Not = ! [ 𝑢 𝑣 𝑤]≡ [ 𝑢′ 𝑣′ 𝑤′ ]↔ [ 𝑢 𝑣 𝑤]= 𝜆 [ 𝑢′ 𝑣′ 𝑤′ ] 𝜆≠ 0 Triple / Equivalent Double / Equals
Perspective Transformations Set bottom row to not [0,0,1] Called a perspective/projective transformation or a homography [ 𝑥′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤] Can compute [x’,y’,w’] via matrix multiplication. How do we get a 2D point? (x’/w’, y’/w’)
Perspective Transformations Set bottom row to not [0,0,1] Called a perspective/projective transformation or a homography [ 𝑥′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤] How many degrees of freedom?
How Many Degrees of Freedom? Can always scale coordinate by non-zero value [ 𝑥 ′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤] Perspective [ 𝑥 ′ 𝑦 ′ 𝑤 ′]≡ 1 𝑖 [ 𝑥′ 𝑦 ′ 𝑤′ ] Homography can always be re-scaled by λ≠0 Typically pick it so last entry is 1. ≡ 1 𝑖 [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤 ]
What’s Preserved With Perspective After multiplication by T (irrespective of T) • Origin is origin: 0 = T0 • Lines are lines • Parallel lines are parallel • Ratios between distances [ 𝑥′ 𝑦 ′ 1 ]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 1 ]≡ 𝑻 [ 𝑥 𝑦 1 ]
Transformation Families In general: transformations are a nested set of groups Diagram credit: R. Szeliski
What Can Homographies Do? Homography example 1: any two views of a planar surface Figure Credit: S. Lazebnik
What Can Homographies Do? Homography example 2: any images from two cameras sharing a camera center Figure Credit: S. Lazebnik
What Can Homographies Do? Homography sort of example “3”: far away scene that can be approximated by a plane Figure credit: Brown & Lowe
Fun With Homographies Original image St. Petersburg photo by A. Tikhonov Virtual camera rotations Slide Credit: A. Efros
Analyzing Patterns Homography Automatically rectified floor The floor (enlarged) Slide from A. Criminisi
Analyzing Patterns Slide from A. Criminisi Automatic rectification From Martin Kemp The Science of Art (manual reconstruction)
Fitting Transformations Setup: have pairs of correspondences (𝑥𝑖 , 𝑦𝑖) (𝑥 ′𝑖 , 𝑦 ′𝑖) M,t [𝑥𝑖 ′ 𝑦𝑖 ′ ]=𝑴 [𝑥𝑖 𝑦𝑖 ]+𝒕 Slide Credit: S. Lazebnik
Fitting Transformation Data: (xi,yi,x’i,y’i) for i=1,…,k Model: [x’i,y’i] = M[xi,yi]+t Objective function: ||[x’i,y’i] – (M[xi,yi]+t)||2 M,t Affine Transformation: M,t
Fitting Transformations [ ⋮ 𝑥𝑖 ′ 𝑦𝑖 ′ ⋮ ]= [ ⋯ ¿ ¿ 𝑥𝑖 𝑦 𝑖 0 0 0 0 𝑥𝑖 𝑦𝑖 1 0 0 1 ¿ ⋯ ][ 𝑚1 𝑚2 𝑚3 𝑚4 𝑡𝑥 𝑡𝑦 ] [𝑥𝑖 ′ 𝑦𝑖 ′ ]= [𝑚1 𝑚2 𝑚3 𝑚4 ][𝑥𝑖 𝑦𝑖 ]+ [𝑡𝑥 𝑡𝑦 ] Given correspondences: [x’i,y’i] ↔ [xi,yi] Set up two equations per point
Fitting Transformations [ ⋮ 𝑥𝑖 ′ 𝑦𝑖 ′ ⋮ ]= [ ⋯ ¿ ¿ 𝑥𝑖 𝑦 𝑖 0 0 0 0 𝑥𝑖 𝑦𝑖 1 0 0 1 ¿ ⋯ ][ 𝑚1 𝑚2 𝑚3 𝑚4 𝑡𝑥 𝑡𝑦 ] 2 equations per point, 6 unknowns How many points do we need to properly constrain the problem? 2k 6
Fitting Transformations [ ⋮ 𝑥𝑖 ′ 𝑦𝑖 ′ ⋮ ]= [ ⋯ ¿ ¿ 𝑥𝑖 𝑦 𝑖 0 0 0 0 𝑥𝑖 𝑦𝑖 1 0 0 1 ¿ ⋯ ][ 𝑚1 𝑚2 𝑚3 𝑚4 𝑡𝑥 𝑡𝑦 ] Want: b = Ax (x contains all parameters) Overconstrained, so solve How? 2k 6 b A x
Fitting Transformation Data: (xi,yi,x’i,y’i) for i=1,…,k Model: [x’i,y’i,1] ≡ H[xi,yi,1] Objective function: It’s complicated H Homography: H
9 k points → 2k 𝑨𝒉=𝟎 What do we use from last time? h ∗ =arg min ‖h‖=1 ‖ h 𝐴 ‖ 2 Eigenvector of AT A with smallest eigenvalue 𝒑 𝒊= [ 𝑥𝑖 𝑦𝑖 1 ] Fitting Transformation [ 𝟎 𝑇 − 𝒑1 𝑇 𝑦1 ′ 𝒑1 𝑇 𝒑1 𝑇 𝟎 𝑇 − 𝑥1 ′ 𝒑1 𝑇 ⋮ 𝟎𝑇 − 𝒑𝑛 𝑇 𝑦𝑛 ′ 𝒑𝑛 𝑇 𝒑𝑛 𝑇 𝟎 𝑇 − 𝑥𝑛 ′ 𝒑𝑛 𝑇 ][ 𝒉𝟏 𝒉𝟐 𝒉𝟑 ]=𝟎 Row 1 of H
In Practice [ 𝟎 𝑇 − 𝒑1 𝑇 𝑦1 ′ 𝒑1 𝑇 𝒑1 𝑇 𝟎 𝑇 − 𝑥1 ′ 𝒑1 𝑇 ⋮ 𝟎𝑇 − 𝒑𝑛 𝑇 𝑦𝑛 ′ 𝒑𝑛 𝑇 𝒑𝑛 𝑇 𝟎 𝑇 − 𝑥𝑛 ′ 𝒑𝑛 𝑇 ][ 𝒉𝟏 𝒉𝟐 𝒉𝟑 ]=𝟎 9 k points → 2k 𝑨𝒉=𝟎 Should consist of lots of {x,y,x’,y’,0, and 1}. If it fails, assume you mistyped. Re-type differently and compare all entries. Debug first with transformations you know. 𝒑 𝒊= [ 𝑥𝑖 𝑦𝑖 1 ] Row 1 of H
Small Nagging Detail ||Ah||2 doesn’t measure model fit (it’s an algebraic error that’s mainly just convenient to minimize) Also, there’s a least-squares setup that’s wrong but often works. ∑ 𝑖=1 𝑘 ‖[𝑥𝑖 ′ , 𝑦𝑖 ′ ]−𝑇 ([𝑥𝑖 , 𝑦𝑖])‖ 2 +‖[𝑥𝑖 , 𝑦𝑖]−𝑇 −1 ([𝑥𝑖 ′ , 𝑦𝑖 ′ ])‖ 2 Really want geometric error:
Small Nagging Detail In RANSAC, we always take just enough points to fit. Why might this not make a big difference when fitting a model with RANSAC? Solution: initialize with algebraic (min ||Ah||), optimize with geometric using standard non-linear optimizer
Image Warping x y x y f(x,y) g(x,y) T(x,y) Given a coordinate transform (x’,y’) = T(x,y) and a source image f(x,y), how do we compute a transformed image g(x’,y’) = f(T(x,y))? Slide Credit: A. Efros
Forward Warping x y x' y' f(x,y) g(x’,y’) T(x,y) Send the value at each pixel (x,y) to the new pixel (x’,y’) = T([x,y]) Slide Credit: A. Efros
Forward Warping x y f(x,y) x-1 x x+1 y-1 y y+1 x'-1 x' x'+1 y'-1 y' y'+1 x' y’ g(x’,y’) If you don’t hit an exact pixel, give the value to each of the neighboring pixels (“splatting”). T(x,y)
Forward Warping Suppose T(x,y) scales by a factor of 3. Hmmmm.
Inverse Warping x y x' y' f(x,y) g(x’,y’) T-1 (x,y) Find out where each pixel g(x’,y’) should get its value from, and steal it. Note: requires ability to invert T Slide Credit: A. Efros
Inverse Warping x'-1 x' x'+1 y'-1 y' y'+1 x' y’ g(x’,y’) x y f(x,y) x-1 x x+1 y-1 y y+1 If you don’t hit an exact pixel, figure out how to take it from the neighbors. T-1 (x,y)
Mosaicing Warped Input 1 I1 Warped Input 2 I2 Image Credit: A. Efros Can warp an image. Pixels that don’t have a corresponding pixel in the image are set to a chosen value (often 0)
Mosaicing Warped Input 1 I1 α Warped Input 2 I2 αI1 + (1-α)I2 Image Credit: A. Efros
Mosaicing Warped Input 1 I1 α Warped Input 2 I2 αI1 + (1-α)I2 Slide Credit: A. Efros Can also warp an image containing 1s. Pixels that don’t have a corresponding pixel in the image are set to a chosen value (often 0)
Putting it Together How do you make a panorama? Step 1: Find “features” to match Step 2: Describe Features Step 3: Match by Nearest Neighbor Step 4: Fit H via RANSAC Step 5: Blend Images
Putting It Together 1 • (Multi-scale) Harris; or • Laplacian of Gaussian Find corners/blobs
Putting It Together 2 Describe Regions Near Features Build histogram of gradient orientations (SIFT) (But in practice use opencv) 𝑥𝑞 ∈ 𝑅128
Putting It Together 3 Match Features Based On Region 𝑥1 ∈ 𝑅128 𝑥2 ∈ 𝑅128 𝑥𝑞 ∈ 𝑅128 𝑥𝑞 Sort by distance to: ‖𝑥𝑞 − 𝑥1‖<‖𝑥𝑞 − 𝑥2 ‖<‖𝑥𝑞 − 𝑥3‖ Accept match if: ‖𝑥𝑞 − 𝑥1‖/‖𝑥𝑞 − 𝑥2‖ Nearest neighbor is far closer than 2nd nearest neighbor
Putting It Together 4 Fit transformation H via RANSAC for trial in range(Ntrials): Pick sample Fit model Check if more inliers Re-fit model with most inliers arg min ‖𝒉‖=1 ‖𝑨𝒉‖ 2
Putting It Together 5 Warp images together Resample images with inverse warping and blend (but in practice, just call opencv for inverse warping)
transformations2 and fitting on image processing.pptx
Backup
A pencil of rays contains all views real camera synthetic camera Can generate any synthetic camera view as long as it has the same center of projection! Slide Credit: A. Efros
Bonus Art
Automatically rectified floor St. Lucy Altarpiece, D. Veneziano Analyzing Patterns What is the (complicated) shape of the floor pattern? Slide from A. Criminisi
From Martin Kemp, The Science of Art (manual reconstruction) Automatic rectification Analyzing Patterns Slide from A. Criminisi
Homography Derivation • This has gotten cut in favor of showing more of the setup. • The key to the set-up is to try to move towards a setup where you can pull [h1,h2,h3] out, or where each row is a linear equation in [h1,h2,h3]
[ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥𝑖 𝑦 𝑖 𝑤𝑖 ]≡ [ 𝑥𝑖 ′ 𝑦𝑖 ′ 𝑤𝑖 ′ ]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥𝑖 𝑦𝑖 𝑤𝑖 ] Want: Recall: 𝒂≡𝒃 𝒂=𝜆𝒃 𝒂×𝒃=𝟎 In turn 𝒑 𝒊= [ 𝑥𝑖 𝑦𝑖 1 ] 𝑯 𝒑𝒊≡ [ 𝒉𝟏 𝑻 𝒉𝟐 𝑻 𝒉𝟑 𝑻 ]𝒑𝒊 ≡ [ 𝒉𝟏 𝑻 𝒑𝒊 𝒉𝟐 𝑻 𝒑𝒊 𝒉𝟑 𝑻 𝒑𝒊 ] [ 𝑥𝑖 ′ 𝑦𝑖 ′ 𝑤𝑖 ′ ]× [ 𝒉𝟏 𝑻 𝒑𝒊 𝒉𝟐 𝑻 𝒑𝒊 𝒉𝟑 𝑻 𝒑𝒊 ]=𝟎 In the end want: Why Cross products? Cross products have explicit forms
Fitting Transformation [ 𝑥𝑖 ′ 𝑦𝑖 ′ 𝑤𝑖 ′ ]× [ 𝒉𝟏 𝑻 𝒑𝒊 𝒉𝟐 𝑻 𝒑𝒊 𝒉𝟑 𝑻 𝒑𝒊 ]=𝟎 Want: [ 𝑦𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 −𝑤𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 𝑤𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 − 𝑥𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 𝑥𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 − 𝑦𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 ]=𝟎 Cross- product [ 𝒉𝟏 𝑻 𝟎−𝑤𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊+ 𝑦𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 𝑤𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝒉𝟐 𝑻 𝟎− 𝑥𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 − 𝑦𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝑥𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 +𝒉𝟑 𝑻 𝟎]=𝟎 Re-arrange and put 0s in Note: calculate this explicitly. It looks ugly, but do it by doing [a,b,c] x [a’,b’,c’] then re-substituting. You want to be able to right- multiply by [h1,h2,h3]
Fitting Transformation [ 𝒉𝟏 𝑻 𝟎−𝑤𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊+ 𝑦𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 𝑤𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝒉𝟐 𝑻 𝟎− 𝑥𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 − 𝑦𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝑥𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 +𝒉𝟑 𝑻 𝟎]=𝟎 Equation Pull out h [ 𝟎 𝑻 −𝑤 ′ 𝑖 𝒑𝒊 𝑻 𝑦 ′𝑖 𝒑𝒊 𝑻 𝑤𝑖 ′ 𝒑𝒊 𝑻 𝟎 𝑻 − 𝑥𝑖 ′ 𝒑𝒊 𝑻 − 𝑦𝑖 ′ 𝒑𝒊 𝑻 𝑥𝑖 ′ 𝒑𝒊 𝑻 𝟎 𝑻 ][ 𝒉𝟏 𝒉𝟐 𝒉𝟑 ]=𝟎 Only two linearly independent equations Yank out h once you have all the coefficients. If you’re head-scratching about the two equations, it’s not obvious to me at first glance that the three equations aren’t linearly independent either.
Simplification: Two-band Blending • Brown & Lowe, 2003 • Only use two bands: high freq. and low freq. • Blend low freq. smoothly • Blend high freq. with no smoothing: binary alpha Figure Credit: Brown & Lowe
Low frequency (l > 2 pixels) High frequency (l < 2 pixels) 2-band “Laplacian Stack” Blending
Linear Blending
2-band Blending

More Related Content

PDF
L5_Bilinear Transformation.pdf
temp2tempa
 
PDF
Visualization of Ordinary Differential equation class lecture
TokeySifullah
 
PPTX
Computer Vision transformations
Wael Badawy
 
PPTX
GraphTransformations.pptx
SrideviNagarjuna
 
PPTX
lec07_transformations.pptx
AneesAbbasi14
 
PPTX
M2L6 Transformations of Functions
mooca76
 
PPT
september4.ppt
CharlesMatu2
 
DOCX
MAT365AffineTransformations
Banks Osborne
 
L5_Bilinear Transformation.pdf
temp2tempa
 
Visualization of Ordinary Differential equation class lecture
TokeySifullah
 
Computer Vision transformations
Wael Badawy
 
GraphTransformations.pptx
SrideviNagarjuna
 
lec07_transformations.pptx
AneesAbbasi14
 
M2L6 Transformations of Functions
mooca76
 
september4.ppt
CharlesMatu2
 
MAT365AffineTransformations
Banks Osborne
 

Similar to transformations2 and fitting on image processing.pptx (20)

PPTX
07 cie552 image_mosaicing
Elsayed Hemayed
 
PPTX
3 d scaling and translation in homogeneous coordinates
KRIPA SHNAKAR TIWARI
 
PPTX
Transformations of functions
KyleJohnson54657
 
PPT
2 d transformation
Ankit Garg
 
PPTX
Numerical Analysis and Its application to Boundary Value Problems
Gobinda Debnath
 
PPTX
Transformations of functions
KyleJohnson54657
 
PPT
2D-Transformations-Transformations are the operations applied to geometrical ...
BINJAD1
 
PPTX
Linear regression, costs & gradient descent
Revanth Kumar
 
PPTX
Math Research.pptx
abdelmalakThabet
 
DOCX
doc
Siddharth Pujari
 
PPTX
for as level graph and transformation topic
bbrise1985
 
PPT
Function transformations
Terry Gastauer
 
PPT
Computer graphics
Bala Murali
 
PDF
Ecuaciones lineal y homogena..
CarlisNuez
 
PDF
A Non Local Boundary Value Problem with Integral Boundary Condition
IJMERJOURNAL
 
PDF
Analysis of a self-sustained vibration of mass-spring oscillator on moving belt
Varun Jadhav
 
PPTX
1-LIMIT-OF-A-FUNCTION.pptx
LimGabriellMontecina
 
PPTX
Laplace & Inverse Transform convoltuion them.pptx
jyotidighole2
 
PDF
Lesson10 transformational geometry
AcceleratedClasses
 
PDF
B.Tech-II_Unit-V
Kundan Kumar
 
07 cie552 image_mosaicing
Elsayed Hemayed
 
3 d scaling and translation in homogeneous coordinates
KRIPA SHNAKAR TIWARI
 
Transformations of functions
KyleJohnson54657
 
2 d transformation
Ankit Garg
 
Numerical Analysis and Its application to Boundary Value Problems
Gobinda Debnath
 
Transformations of functions
KyleJohnson54657
 
2D-Transformations-Transformations are the operations applied to geometrical ...
BINJAD1
 
Linear regression, costs & gradient descent
Revanth Kumar
 
Math Research.pptx
abdelmalakThabet
 
doc
Siddharth Pujari
 
for as level graph and transformation topic
bbrise1985
 
Function transformations
Terry Gastauer
 
Computer graphics
Bala Murali
 
Ecuaciones lineal y homogena..
CarlisNuez
 
A Non Local Boundary Value Problem with Integral Boundary Condition
IJMERJOURNAL
 
Analysis of a self-sustained vibration of mass-spring oscillator on moving belt
Varun Jadhav
 
1-LIMIT-OF-A-FUNCTION.pptx
LimGabriellMontecina
 
Laplace & Inverse Transform convoltuion them.pptx
jyotidighole2
 
Lesson10 transformational geometry
AcceleratedClasses
 
B.Tech-II_Unit-V
Kundan Kumar
 
Ad

More from Indra Hermawan (20)

PDF
L6 Visual Output LED_7SEGMEN_LEDMATRIX upate.pdf
Indra Hermawan
 
PDF
Organic Light Emitting Diode (OLED) for embedded system
Indra Hermawan
 
PDF
Organic Light Emitting Diode (OLED) in embedded system
Indra Hermawan
 
PDF
L6 Visual Output LED_7SEGMEN_LEDMATRIX upate.pdf
Indra Hermawan
 
PPTX
over or under fitting in neural network.pptx
Indra Hermawan
 
PPT
zigbee networks using xbee modules for wsn
Indra Hermawan
 
PDF
pengenalan mengnai ai dan machine learning
Indra Hermawan
 
PPTX
Market_Behavior_Analytics_AIoT TMJ 6A.pptx
Indra Hermawan
 
PPTX
7-1-Artificial Neural Network dan multilayer perceptron.pptx
Indra Hermawan
 
PPTX
ZigBee_Presentation sensor and cotrol network
Indra Hermawan
 
PPTX
ZigBee_Comprehensive_Presentation and implementaiton
Indra Hermawan
 
PPTX
Numerical Linear Algebra in digital image processing
Indra Hermawan
 
PPTX
Filtering in digital signal and image processing
Indra Hermawan
 
PPTX
Sistem Pengomposan Cerdas Berbasis IoT untuk Pengelolaan Sampah Organik Berke...
Indra Hermawan
 
PDF
introduction to camera, the operation of camera
Indra Hermawan
 
PDF
lab_linear_regression_hy539 (1)_221109_035050.pdf
Indra Hermawan
 
PPTX
perancangan program kerja KBK perancangan program kerja KBK perancangan progr...
Indra Hermawan
 
PPT
zigbee networks using xbee modules zigbee networks using xbee modules
Indra Hermawan
 
PPTX
lecture_8-wsn.pptx
Indra Hermawan
 
PPT
Trik Pembuatan Surat Lamaran Pekerjaan.ppt
Indra Hermawan
 
L6 Visual Output LED_7SEGMEN_LEDMATRIX upate.pdf
Indra Hermawan
 
Organic Light Emitting Diode (OLED) for embedded system
Indra Hermawan
 
Organic Light Emitting Diode (OLED) in embedded system
Indra Hermawan
 
L6 Visual Output LED_7SEGMEN_LEDMATRIX upate.pdf
Indra Hermawan
 
over or under fitting in neural network.pptx
Indra Hermawan
 
zigbee networks using xbee modules for wsn
Indra Hermawan
 
pengenalan mengnai ai dan machine learning
Indra Hermawan
 
Market_Behavior_Analytics_AIoT TMJ 6A.pptx
Indra Hermawan
 
7-1-Artificial Neural Network dan multilayer perceptron.pptx
Indra Hermawan
 
ZigBee_Presentation sensor and cotrol network
Indra Hermawan
 
ZigBee_Comprehensive_Presentation and implementaiton
Indra Hermawan
 
Numerical Linear Algebra in digital image processing
Indra Hermawan
 
Filtering in digital signal and image processing
Indra Hermawan
 
Sistem Pengomposan Cerdas Berbasis IoT untuk Pengelolaan Sampah Organik Berke...
Indra Hermawan
 
introduction to camera, the operation of camera
Indra Hermawan
 
lab_linear_regression_hy539 (1)_221109_035050.pdf
Indra Hermawan
 
perancangan program kerja KBK perancangan program kerja KBK perancangan progr...
Indra Hermawan
 
zigbee networks using xbee modules zigbee networks using xbee modules
Indra Hermawan
 
lecture_8-wsn.pptx
Indra Hermawan
 
Trik Pembuatan Surat Lamaran Pekerjaan.ppt
Indra Hermawan
 
Ad

Recently uploaded (20)

PPTX
办理HFM文凭|购买代特莫尔德音乐学院毕业证文凭100%复刻安全可靠的
1cz3lou8
 
PPTX
Mobile-Device-Management-MDM-Architecture.pptx
pranavnandwanshi99
 
PPTX
西班牙海牙认证瓦伦西亚国际大学毕业证与成绩单文凭复刻快速办理毕业证书
sw6vvn9s
 
PDF
Portable Veterinary Ultrasound Scanners & Animal Medical Equipment - TcCryo
3447752272
 
PPTX
Normal distriutionvggggggggggggggggggg.pptx
JayeshTaneja4
 
PPTX
great itemsgreat itemsgreat itemsgreat items.pptx
saurabh13smr
 
PPTX
atoma.pptxejejejejeejejjeejeejeju3u3u3u3
manthan912009
 
PPT
3 01032017tyuiryhjrhyureyhjkfdhghfrugjhf
DharaniMani4
 
PPTX
Basics of Memristors and fundamentals.pptx
onterusmail
 
PDF
Endalamaw Kebede.pdfvvbhjjnhgggftygtttfgh
SirajudinAkmel1
 
PPTX
原版UMiami毕业证文凭迈阿密大学学费单定制学历在线制作硕士毕业证
jicaaeb0
 
PPTX
13. ANAESTHETICS AND ALCOHOLS.pptx fucking
sriramraja650
 
PPTX
Modern machinery.pptx sjsjnshhsnsnnjnnbbbb
raipureastha08
 
PPT
community diagnosis slides show health. ppt
michaelbrucebwana
 
PPTX
cocomo-220726173706-141e08f0.tyuiuuupptx
DharaniMani4
 
PPTX
Basics of Memristors from zero to hero.pptx
onterusmail
 
PPT
Susunan & Bagian DRAWING 153UWYHSGDGH.ppt
RezaFbriadi
 
PPTX
INTERNET OF THINGS (IOT) network of interconnected devices.
rp1256748
 
PPTX
G6Q1 WEEK 2 SCIENCE PPT.pptxLVLLLLLLLLLLLLLLLLL
DitaSIdnay
 
PPTX
PPT on the topic of programming language
dishasindhava
 
办理HFM文凭|购买代特莫尔德音乐学院毕业证文凭100%复刻安全可靠的
1cz3lou8
 
Mobile-Device-Management-MDM-Architecture.pptx
pranavnandwanshi99
 
西班牙海牙认证瓦伦西亚国际大学毕业证与成绩单文凭复刻快速办理毕业证书
sw6vvn9s
 
Portable Veterinary Ultrasound Scanners & Animal Medical Equipment - TcCryo
3447752272
 
Normal distriutionvggggggggggggggggggg.pptx
JayeshTaneja4
 
great itemsgreat itemsgreat itemsgreat items.pptx
saurabh13smr
 
atoma.pptxejejejejeejejjeejeejeju3u3u3u3
manthan912009
 
3 01032017tyuiryhjrhyureyhjkfdhghfrugjhf
DharaniMani4
 
Basics of Memristors and fundamentals.pptx
onterusmail
 
Endalamaw Kebede.pdfvvbhjjnhgggftygtttfgh
SirajudinAkmel1
 
原版UMiami毕业证文凭迈阿密大学学费单定制学历在线制作硕士毕业证
jicaaeb0
 
13. ANAESTHETICS AND ALCOHOLS.pptx fucking
sriramraja650
 
Modern machinery.pptx sjsjnshhsnsnnjnnbbbb
raipureastha08
 
community diagnosis slides show health. ppt
michaelbrucebwana
 
cocomo-220726173706-141e08f0.tyuiuuupptx
DharaniMani4
 
Basics of Memristors from zero to hero.pptx
onterusmail
 
Susunan & Bagian DRAWING 153UWYHSGDGH.ppt
RezaFbriadi
 
INTERNET OF THINGS (IOT) network of interconnected devices.
rp1256748
 
G6Q1 WEEK 2 SCIENCE PPT.pptxLVLLLLLLLLLLLLLLLLL
DitaSIdnay
 
PPT on the topic of programming language
dishasindhava
 

transformations2 and fitting on image processing.pptx

  • 1. Transformation s and Fitting EECS 442 – David Fouhey Winter 2023, University of Michigan https://web.eecs.umich.edu/~fouhey/teaching/EECS442_W23/
  • 2. Administrivia • Discussion this week = office hours
  • 3. So Far 1. How do we find distinctive / easy to locate features? (Harris/Laplacian of Gaussian) 2. How do we describe the regions around them? (histogram of gradients) 3. How do we match features? (L2 distance) 4. How do we handle outliers? (RANSAC)
  • 4. Today As promised: warping one image to another
  • 5. Why Mosaic? • Compact Camera FOV = 50 x 35° Slide credit: Brown & Lowe
  • 6. Why Mosaic? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135° Slide credit: Brown & Lowe
  • 7. Why Mosaic? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135° • Panoramic Mosaic = 360 x 180° Slide credit: Brown & Lowe
  • 8. Why Bother With This Math? Slide credit: A. Efros
  • 9. Homework 1 Style Translation only via alignment Slide credit: A. Efros
  • 11. Image Transformations f x T g x f x T g x Image filtering: change range of image 𝑔( 𝑥)=𝑇 ( 𝑓 ( 𝑥)) 𝑔 ( 𝑥 )= 𝑓 ¿ Image warping: change domain of image Slide credit: A. Efros
  • 12. Image Transformations T T Image filtering: change range of image 𝑔(𝑥 , 𝑦)=𝑇 ( 𝑓 (𝑥, 𝑦 )) 𝑔 ( 𝑥 , 𝑦 )= 𝑓 ¿ Image warping: change domain of image f g f g Slide credit: A. Efros
  • 13. Parametric (Global) warping translation rotation aspect affine perspective cylindrical Examples of parametric warps Slide credit: A. Efros
  • 14. Parametric (Global) Warping T p’ = (x’,y’) T is a coordinate changing machine p = (x,y) Note: T is the same for all points, has relatively few parameters, and does not depend on image content 𝒑′ =𝑇 (𝒑) Slide credit: A. Efros
  • 15. Parametric (Global) Warping T p’ = (x’,y’) p = (x,y) Today we’ll deal with linear warps 𝒑′ ≡𝑻𝒑 T: matrix; p, p’: 2D points. Start with normal points and =, then do homogeneous cords and ≡ Slide credit: A. Efros
  • 16. Scaling  2 Scaling multiplies each component (x,y) by a scalar. Uniform scaling is the same for all components. Note the corner goes from (1,1) to (2,2) Slide credit: A. Efros
  • 17. Scaling Non-uniform scaling multiplies each component by a different scalar. X  2, Y  0.5 Slide credit: A. Efros
  • 18. Scaling What does T look like? 𝑥′ =𝑎𝑥 𝑦 ′ =𝑏𝑦 Let’s convert to a matrix: [𝑥 ′ 𝑦 ′ ]= [𝑎 0 0 𝑏][𝑥 𝑦 ] scaling matrix S What’s the inverse of S? Slide credit: A. Efros
  • 19. 2D Rotation Rotation Matrix But wait! Aren’t sin/cos non-linear? x’ is a linear combination/function of x, y x’ is not a linear function of θ What’s the inverse of Rθ? 𝑰 =𝑹𝜽 𝑇 𝑹𝜽 [𝑥 ′ 𝑦 ′ ]= [c os ⁡(𝜃) − sin (𝜃 ) sin ( 𝜃) cos(𝜃 ) ][𝑥 𝑦 ] Slide credit: A. Efros
  • 20. Things You Can Do With 2x2 Identity / No Transformation Shear [𝑥 ′ 𝑦 ′ ]= [ 1 𝑠 h𝑥 𝑠h𝑦 1 ][𝑥 𝑦 ] [𝑥 ′ 𝑦 ′ ]= [1 0 0 1 ][𝑥 𝑦 ] Slide credit: A. Efros
  • 21. Things You Can Do With 2x2 2D Mirror About Y-Axis [𝑥 ′ 𝑦 ′ ]= [−1 0 0 1 ][𝑥 𝑦 ] Before After 2D Mirror About X,Y [𝑥 ′ 𝑦 ′ ]= [−1 0 0 −1][𝑥 𝑦 ] Before After Slide credit: A. Efros
  • 22. What’s Preserved? Projections of parallel 3D lines are not necessarily parallel, so not parallelism 3D lines project to 2D lines so lines are preserved Distant objects are smaller so size is not preserved
  • 23. What’s Preserved With a 2x2 [𝑥 ′ 𝑦 ′ ]= [𝑎 𝑏 𝑐 𝑑][𝑥 𝑦 ]=𝑇 [𝑥 𝑦] After multiplication by T (irrespective of T) • Origin is origin: 0 = T0 • Lines are lines • Parallel lines are parallel
  • 24. Things You Can’t Do With 2x2 What about translation? x’ = x + tx, y’ = y+ty +(2,2) How do we make it linear?
  • 25. Homogeneous Coordinates Again What about translation? x’ = x + tx, y’ = y+ty +(2,2) [ 𝑥+𝑡𝑥 𝑦 +𝑡 𝑦 1 ]≡ [ 𝑥′ 𝑦 ′ 1 ]≡ [ 1 0 𝑡𝑥 0 1 𝑡𝑦 0 0 1 ][ 𝑥 𝑦 1 ] Slide credit: A. Efros
  • 26. Representing 2D Transformations How do we represent a 2D transformation? Let’s pick scaling [ 𝑥′ 𝑦 ′ 1 ]≡ [ 𝑠𝑥 0 𝑎 0 𝑠𝑦 𝑏 𝑑 𝑒 𝑓 ][ 𝑥 𝑦 1 ] a b d e f 0 0 0 0 1 What’s
  • 27. Affine Transformations Affine: linear transformation plus translation In general (without homogeneous coordinates) 𝒙 ′= 𝑨𝒙+𝒃 Will the last coordinate w’ always be 1? [ 𝑥 ′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 0 0 1 ][ 𝑥 𝑦 1 ] t
  • 28. Matrix Composition [ 𝑥′ 𝑦 ′ 𝑤 ′]≡ [ 1 0 𝑡𝑥 0 1 𝑡 𝑦 0 0 1 ][ cos (𝜃 ) −sin ( 𝜃) 0 sin (𝜃) cos ( 𝜃) 0 0 0 1][ 𝑠𝑥 0 0 0 𝑠 𝑦 0 0 0 1 ][ 𝑥 𝑦 𝑤] 𝑇 (𝑡𝑥 ,𝑡𝑦 ) 𝑅( 𝜃) 𝑆(𝑠𝑥 ,𝑠𝑦 ) We can combine transformations via matrix multiplication. Does order matter? Slide credit: A. Efros
  • 29. What’s Preserved With Affine After multiplication by T (irrespective of T) • Origin is origin: 0 = T0 • Lines are lines • Parallel lines are parallel [ 𝑥′ 𝑦 ′ 1 ]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 0 0 1 ][ 𝑥 𝑦 1 ]≡ 𝑻 [ 𝑥 𝑦 1 ]
  • 30. Homogeneous Equivalence z x y [x,y,w] λ[x,y,w] Two homogeneous coordinates are equivalent if they are proportional to each other. Not = ! [ 𝑢 𝑣 𝑤]≡ [ 𝑢′ 𝑣′ 𝑤′ ]↔ [ 𝑢 𝑣 𝑤]= 𝜆 [ 𝑢′ 𝑣′ 𝑤′ ] 𝜆≠ 0 Triple / Equivalent Double / Equals
  • 31. Perspective Transformations Set bottom row to not [0,0,1] Called a perspective/projective transformation or a homography [ 𝑥′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤] Can compute [x’,y’,w’] via matrix multiplication. How do we get a 2D point? (x’/w’, y’/w’)
  • 32. Perspective Transformations Set bottom row to not [0,0,1] Called a perspective/projective transformation or a homography [ 𝑥′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤] How many degrees of freedom?
  • 33. How Many Degrees of Freedom? Can always scale coordinate by non-zero value [ 𝑥 ′ 𝑦 ′ 𝑤 ′]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤] Perspective [ 𝑥 ′ 𝑦 ′ 𝑤 ′]≡ 1 𝑖 [ 𝑥′ 𝑦 ′ 𝑤′ ] Homography can always be re-scaled by λ≠0 Typically pick it so last entry is 1. ≡ 1 𝑖 [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 𝑤 ]
  • 34. What’s Preserved With Perspective After multiplication by T (irrespective of T) • Origin is origin: 0 = T0 • Lines are lines • Parallel lines are parallel • Ratios between distances [ 𝑥′ 𝑦 ′ 1 ]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥 𝑦 1 ]≡ 𝑻 [ 𝑥 𝑦 1 ]
  • 35. Transformation Families In general: transformations are a nested set of groups Diagram credit: R. Szeliski
  • 36. What Can Homographies Do? Homography example 1: any two views of a planar surface Figure Credit: S. Lazebnik
  • 37. What Can Homographies Do? Homography example 2: any images from two cameras sharing a camera center Figure Credit: S. Lazebnik
  • 38. What Can Homographies Do? Homography sort of example “3”: far away scene that can be approximated by a plane Figure credit: Brown & Lowe
  • 39. Fun With Homographies Original image St. Petersburg photo by A. Tikhonov Virtual camera rotations Slide Credit: A. Efros
  • 40. Analyzing Patterns Homography Automatically rectified floor The floor (enlarged) Slide from A. Criminisi
  • 41. Analyzing Patterns Slide from A. Criminisi Automatic rectification From Martin Kemp The Science of Art (manual reconstruction)
  • 42. Fitting Transformations Setup: have pairs of correspondences (𝑥𝑖 , 𝑦𝑖) (𝑥 ′𝑖 , 𝑦 ′𝑖) M,t [𝑥𝑖 ′ 𝑦𝑖 ′ ]=𝑴 [𝑥𝑖 𝑦𝑖 ]+𝒕 Slide Credit: S. Lazebnik
  • 43. Fitting Transformation Data: (xi,yi,x’i,y’i) for i=1,…,k Model: [x’i,y’i] = M[xi,yi]+t Objective function: ||[x’i,y’i] – (M[xi,yi]+t)||2 M,t Affine Transformation: M,t
  • 44. Fitting Transformations [ ⋮ 𝑥𝑖 ′ 𝑦𝑖 ′ ⋮ ]= [ ⋯ ¿ ¿ 𝑥𝑖 𝑦 𝑖 0 0 0 0 𝑥𝑖 𝑦𝑖 1 0 0 1 ¿ ⋯ ][ 𝑚1 𝑚2 𝑚3 𝑚4 𝑡𝑥 𝑡𝑦 ] [𝑥𝑖 ′ 𝑦𝑖 ′ ]= [𝑚1 𝑚2 𝑚3 𝑚4 ][𝑥𝑖 𝑦𝑖 ]+ [𝑡𝑥 𝑡𝑦 ] Given correspondences: [x’i,y’i] ↔ [xi,yi] Set up two equations per point
  • 45. Fitting Transformations [ ⋮ 𝑥𝑖 ′ 𝑦𝑖 ′ ⋮ ]= [ ⋯ ¿ ¿ 𝑥𝑖 𝑦 𝑖 0 0 0 0 𝑥𝑖 𝑦𝑖 1 0 0 1 ¿ ⋯ ][ 𝑚1 𝑚2 𝑚3 𝑚4 𝑡𝑥 𝑡𝑦 ] 2 equations per point, 6 unknowns How many points do we need to properly constrain the problem? 2k 6
  • 46. Fitting Transformations [ ⋮ 𝑥𝑖 ′ 𝑦𝑖 ′ ⋮ ]= [ ⋯ ¿ ¿ 𝑥𝑖 𝑦 𝑖 0 0 0 0 𝑥𝑖 𝑦𝑖 1 0 0 1 ¿ ⋯ ][ 𝑚1 𝑚2 𝑚3 𝑚4 𝑡𝑥 𝑡𝑦 ] Want: b = Ax (x contains all parameters) Overconstrained, so solve How? 2k 6 b A x
  • 47. Fitting Transformation Data: (xi,yi,x’i,y’i) for i=1,…,k Model: [x’i,y’i,1] ≡ H[xi,yi,1] Objective function: It’s complicated H Homography: H
  • 48. 9 k points → 2k 𝑨𝒉=𝟎 What do we use from last time? h ∗ =arg min ‖h‖=1 ‖ h 𝐴 ‖ 2 Eigenvector of AT A with smallest eigenvalue 𝒑 𝒊= [ 𝑥𝑖 𝑦𝑖 1 ] Fitting Transformation [ 𝟎 𝑇 − 𝒑1 𝑇 𝑦1 ′ 𝒑1 𝑇 𝒑1 𝑇 𝟎 𝑇 − 𝑥1 ′ 𝒑1 𝑇 ⋮ 𝟎𝑇 − 𝒑𝑛 𝑇 𝑦𝑛 ′ 𝒑𝑛 𝑇 𝒑𝑛 𝑇 𝟎 𝑇 − 𝑥𝑛 ′ 𝒑𝑛 𝑇 ][ 𝒉𝟏 𝒉𝟐 𝒉𝟑 ]=𝟎 Row 1 of H
  • 49. In Practice [ 𝟎 𝑇 − 𝒑1 𝑇 𝑦1 ′ 𝒑1 𝑇 𝒑1 𝑇 𝟎 𝑇 − 𝑥1 ′ 𝒑1 𝑇 ⋮ 𝟎𝑇 − 𝒑𝑛 𝑇 𝑦𝑛 ′ 𝒑𝑛 𝑇 𝒑𝑛 𝑇 𝟎 𝑇 − 𝑥𝑛 ′ 𝒑𝑛 𝑇 ][ 𝒉𝟏 𝒉𝟐 𝒉𝟑 ]=𝟎 9 k points → 2k 𝑨𝒉=𝟎 Should consist of lots of {x,y,x’,y’,0, and 1}. If it fails, assume you mistyped. Re-type differently and compare all entries. Debug first with transformations you know. 𝒑 𝒊= [ 𝑥𝑖 𝑦𝑖 1 ] Row 1 of H
  • 50. Small Nagging Detail ||Ah||2 doesn’t measure model fit (it’s an algebraic error that’s mainly just convenient to minimize) Also, there’s a least-squares setup that’s wrong but often works. ∑ 𝑖=1 𝑘 ‖[𝑥𝑖 ′ , 𝑦𝑖 ′ ]−𝑇 ([𝑥𝑖 , 𝑦𝑖])‖ 2 +‖[𝑥𝑖 , 𝑦𝑖]−𝑇 −1 ([𝑥𝑖 ′ , 𝑦𝑖 ′ ])‖ 2 Really want geometric error:
  • 51. Small Nagging Detail In RANSAC, we always take just enough points to fit. Why might this not make a big difference when fitting a model with RANSAC? Solution: initialize with algebraic (min ||Ah||), optimize with geometric using standard non-linear optimizer
  • 52. Image Warping x y x y f(x,y) g(x,y) T(x,y) Given a coordinate transform (x’,y’) = T(x,y) and a source image f(x,y), how do we compute a transformed image g(x’,y’) = f(T(x,y))? Slide Credit: A. Efros
  • 53. Forward Warping x y x' y' f(x,y) g(x’,y’) T(x,y) Send the value at each pixel (x,y) to the new pixel (x’,y’) = T([x,y]) Slide Credit: A. Efros
  • 54. Forward Warping x y f(x,y) x-1 x x+1 y-1 y y+1 x'-1 x' x'+1 y'-1 y' y'+1 x' y’ g(x’,y’) If you don’t hit an exact pixel, give the value to each of the neighboring pixels (“splatting”). T(x,y)
  • 55. Forward Warping Suppose T(x,y) scales by a factor of 3. Hmmmm.
  • 56. Inverse Warping x y x' y' f(x,y) g(x’,y’) T-1 (x,y) Find out where each pixel g(x’,y’) should get its value from, and steal it. Note: requires ability to invert T Slide Credit: A. Efros
  • 57. Inverse Warping x'-1 x' x'+1 y'-1 y' y'+1 x' y’ g(x’,y’) x y f(x,y) x-1 x x+1 y-1 y y+1 If you don’t hit an exact pixel, figure out how to take it from the neighbors. T-1 (x,y)
  • 58. Mosaicing Warped Input 1 I1 Warped Input 2 I2 Image Credit: A. Efros Can warp an image. Pixels that don’t have a corresponding pixel in the image are set to a chosen value (often 0)
  • 59. Mosaicing Warped Input 1 I1 α Warped Input 2 I2 αI1 + (1-α)I2 Image Credit: A. Efros
  • 60. Mosaicing Warped Input 1 I1 α Warped Input 2 I2 αI1 + (1-α)I2 Slide Credit: A. Efros Can also warp an image containing 1s. Pixels that don’t have a corresponding pixel in the image are set to a chosen value (often 0)
  • 61. Putting it Together How do you make a panorama? Step 1: Find “features” to match Step 2: Describe Features Step 3: Match by Nearest Neighbor Step 4: Fit H via RANSAC Step 5: Blend Images
  • 62. Putting It Together 1 • (Multi-scale) Harris; or • Laplacian of Gaussian Find corners/blobs
  • 63. Putting It Together 2 Describe Regions Near Features Build histogram of gradient orientations (SIFT) (But in practice use opencv) 𝑥𝑞 ∈ 𝑅128
  • 64. Putting It Together 3 Match Features Based On Region 𝑥1 ∈ 𝑅128 𝑥2 ∈ 𝑅128 𝑥𝑞 ∈ 𝑅128 𝑥𝑞 Sort by distance to: ‖𝑥𝑞 − 𝑥1‖<‖𝑥𝑞 − 𝑥2 ‖<‖𝑥𝑞 − 𝑥3‖ Accept match if: ‖𝑥𝑞 − 𝑥1‖/‖𝑥𝑞 − 𝑥2‖ Nearest neighbor is far closer than 2nd nearest neighbor
  • 65. Putting It Together 4 Fit transformation H via RANSAC for trial in range(Ntrials): Pick sample Fit model Check if more inliers Re-fit model with most inliers arg min ‖𝒉‖=1 ‖𝑨𝒉‖ 2
  • 66. Putting It Together 5 Warp images together Resample images with inverse warping and blend (but in practice, just call opencv for inverse warping)
  • 69. A pencil of rays contains all views real camera synthetic camera Can generate any synthetic camera view as long as it has the same center of projection! Slide Credit: A. Efros
  • 71. Automatically rectified floor St. Lucy Altarpiece, D. Veneziano Analyzing Patterns What is the (complicated) shape of the floor pattern? Slide from A. Criminisi
  • 72. From Martin Kemp, The Science of Art (manual reconstruction) Automatic rectification Analyzing Patterns Slide from A. Criminisi
  • 73. Homography Derivation • This has gotten cut in favor of showing more of the setup. • The key to the set-up is to try to move towards a setup where you can pull [h1,h2,h3] out, or where each row is a linear equation in [h1,h2,h3]
  • 74. [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥𝑖 𝑦 𝑖 𝑤𝑖 ]≡ [ 𝑥𝑖 ′ 𝑦𝑖 ′ 𝑤𝑖 ′ ]≡ [ 𝑎 𝑏 𝑐 𝑑 𝑒 𝑓 𝑔 h 𝑖 ][ 𝑥𝑖 𝑦𝑖 𝑤𝑖 ] Want: Recall: 𝒂≡𝒃 𝒂=𝜆𝒃 𝒂×𝒃=𝟎 In turn 𝒑 𝒊= [ 𝑥𝑖 𝑦𝑖 1 ] 𝑯 𝒑𝒊≡ [ 𝒉𝟏 𝑻 𝒉𝟐 𝑻 𝒉𝟑 𝑻 ]𝒑𝒊 ≡ [ 𝒉𝟏 𝑻 𝒑𝒊 𝒉𝟐 𝑻 𝒑𝒊 𝒉𝟑 𝑻 𝒑𝒊 ] [ 𝑥𝑖 ′ 𝑦𝑖 ′ 𝑤𝑖 ′ ]× [ 𝒉𝟏 𝑻 𝒑𝒊 𝒉𝟐 𝑻 𝒑𝒊 𝒉𝟑 𝑻 𝒑𝒊 ]=𝟎 In the end want: Why Cross products? Cross products have explicit forms
  • 75. Fitting Transformation [ 𝑥𝑖 ′ 𝑦𝑖 ′ 𝑤𝑖 ′ ]× [ 𝒉𝟏 𝑻 𝒑𝒊 𝒉𝟐 𝑻 𝒑𝒊 𝒉𝟑 𝑻 𝒑𝒊 ]=𝟎 Want: [ 𝑦𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 −𝑤𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 𝑤𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 − 𝑥𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 𝑥𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 − 𝑦𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 ]=𝟎 Cross- product [ 𝒉𝟏 𝑻 𝟎−𝑤𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊+ 𝑦𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 𝑤𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝒉𝟐 𝑻 𝟎− 𝑥𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 − 𝑦𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝑥𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 +𝒉𝟑 𝑻 𝟎]=𝟎 Re-arrange and put 0s in Note: calculate this explicitly. It looks ugly, but do it by doing [a,b,c] x [a’,b’,c’] then re-substituting. You want to be able to right- multiply by [h1,h2,h3]
  • 76. Fitting Transformation [ 𝒉𝟏 𝑻 𝟎−𝑤𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊+ 𝑦𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 𝑤𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝒉𝟐 𝑻 𝟎− 𝑥𝑖 ′ 𝒉𝟑 𝑻 𝒑𝒊 − 𝑦𝑖 ′ 𝒉𝟏 𝑻 𝒑𝒊 +𝑥𝑖 ′ 𝒉𝟐 𝑻 𝒑𝒊 +𝒉𝟑 𝑻 𝟎]=𝟎 Equation Pull out h [ 𝟎 𝑻 −𝑤 ′ 𝑖 𝒑𝒊 𝑻 𝑦 ′𝑖 𝒑𝒊 𝑻 𝑤𝑖 ′ 𝒑𝒊 𝑻 𝟎 𝑻 − 𝑥𝑖 ′ 𝒑𝒊 𝑻 − 𝑦𝑖 ′ 𝒑𝒊 𝑻 𝑥𝑖 ′ 𝒑𝒊 𝑻 𝟎 𝑻 ][ 𝒉𝟏 𝒉𝟐 𝒉𝟑 ]=𝟎 Only two linearly independent equations Yank out h once you have all the coefficients. If you’re head-scratching about the two equations, it’s not obvious to me at first glance that the three equations aren’t linearly independent either.
  • 77. Simplification: Two-band Blending • Brown & Lowe, 2003 • Only use two bands: high freq. and low freq. • Blend low freq. smoothly • Blend high freq. with no smoothing: binary alpha Figure Credit: Brown & Lowe
  • 78. Low frequency (l > 2 pixels) High frequency (l < 2 pixels) 2-band “Laplacian Stack” Blending