CV2_CH9 ppt x chapter wise notes from CV

AartiDadheech 3 views 117 slides Oct 29, 2025
Slide 1
Slide 1 of 210
Slide 1
1
Slide 2
2
Slide 3
3
Slide 4
4
Slide 5
5
Slide 6
6
Slide 7
7
Slide 8
8
Slide 9
9
Slide 10
10
Slide 11
11
Slide 12
12
Slide 13
13
Slide 14
14
Slide 15
15
Slide 16
16
Slide 17
17
Slide 18
18
Slide 19
19
Slide 20
20
Slide 21
21
Slide 22
22
Slide 23
23
Slide 24
24
Slide 25
25
Slide 26
26
Slide 27
27
Slide 28
28
Slide 29
29
Slide 30
30
Slide 31
31
Slide 32
32
Slide 33
33
Slide 34
34
Slide 35
35
Slide 36
36
Slide 37
37
Slide 38
38
Slide 39
39
Slide 40
40
Slide 41
41
Slide 42
42
Slide 43
43
Slide 44
44
Slide 45
45
Slide 46
46
Slide 47
47
Slide 48
48
Slide 49
49
Slide 50
50
Slide 51
51
Slide 52
52
Slide 53
53
Slide 54
54
Slide 55
55
Slide 56
56
Slide 57
57
Slide 58
58
Slide 59
59
Slide 60
60
Slide 61
61
Slide 62
62
Slide 63
63
Slide 64
64
Slide 65
65
Slide 66
66
Slide 67
67
Slide 68
68
Slide 69
69
Slide 70
70
Slide 71
71
Slide 72
72
Slide 73
73
Slide 74
74
Slide 75
75
Slide 76
76
Slide 77
77
Slide 78
78
Slide 79
79
Slide 80
80
Slide 81
81
Slide 82
82
Slide 83
83
Slide 84
84
Slide 85
85
Slide 86
86
Slide 87
87
Slide 88
88
Slide 89
89
Slide 90
90
Slide 91
91
Slide 92
92
Slide 93
93
Slide 94
94
Slide 95
95
Slide 96
96
Slide 97
97
Slide 98
98
Slide 99
99
Slide 100
100
Slide 101
101
Slide 102
102
Slide 103
103
Slide 104
104
Slide 105
105
Slide 106
106
Slide 107
107
Slide 108
108
Slide 109
109
Slide 110
110
Slide 111
111
Slide 112
112
Slide 113
113
Slide 114
114
Slide 115
115
Slide 116
116
Slide 117
117
Slide 118
118
Slide 119
119
Slide 120
120
Slide 121
121
Slide 122
122
Slide 123
123
Slide 124
124
Slide 125
125
Slide 126
126
Slide 127
127
Slide 128
128
Slide 129
129
Slide 130
130
Slide 131
131
Slide 132
132
Slide 133
133
Slide 134
134
Slide 135
135
Slide 136
136
Slide 137
137
Slide 138
138
Slide 139
139
Slide 140
140
Slide 141
141
Slide 142
142
Slide 143
143
Slide 144
144
Slide 145
145
Slide 146
146
Slide 147
147
Slide 148
148
Slide 149
149
Slide 150
150
Slide 151
151
Slide 152
152
Slide 153
153
Slide 154
154
Slide 155
155
Slide 156
156
Slide 157
157
Slide 158
158
Slide 159
159
Slide 160
160
Slide 161
161
Slide 162
162
Slide 163
163
Slide 164
164
Slide 165
165
Slide 166
166
Slide 167
167
Slide 168
168
Slide 169
169
Slide 170
170
Slide 171
171
Slide 172
172
Slide 173
173
Slide 174
174
Slide 175
175
Slide 176
176
Slide 177
177
Slide 178
178
Slide 179
179
Slide 180
180
Slide 181
181
Slide 182
182
Slide 183
183
Slide 184
184
Slide 185
185
Slide 186
186
Slide 187
187
Slide 188
188
Slide 189
189
Slide 190
190
Slide 191
191
Slide 192
192
Slide 193
193
Slide 194
194
Slide 195
195
Slide 196
196
Slide 197
197
Slide 198
198
Slide 199
199
Slide 200
200
Slide 201
201
Slide 202
202
Slide 203
203
Slide 204
204
Slide 205
205
Slide 206
206
Slide 207
207
Slide 208
208
Slide 209
209
Slide 210
210

About This Presentation

chapter wise notes


Slide Content

Motion Estimation Presenter: Yu-Hsiang Shao ( 邵育翔 ) Phone: 0972326030 E-Mail : [email protected] Advisor: Prof. Chiou - Shann Fuh Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R.O.C. Advanced Computer Vision  Chapter 9

Slide credits Most of these slides were adapted from: Ioannis ( Yannis ) Gkioulekas  (16-385 Computer Vision, Carnegie Mellon University ). Delbert Gideon ( SlideServe ). 湯竣淋 (ACV, 2020)

Overview Image processing Geometry-based vision Physics-based vision Semantic vision Dealing with motion Chapter 9

9.1 Translational alignment  9.2 Optical flow. Horn- Schunck flow. (HW3) 9.3 Parametric motion 9.4 Layered motion Overview of this lecture

9.1 Translational Alignment

http://www.humansensing.cs.cmu.edu/intraface/

How can I find in the image?

Template Matching Alignment between two images or image patches A suitable error metric must first be chosen to compare the images Where they are located in

Translational Alignment Minimum of Sum of Squared Difference (SSD) Assumption: corresponding pixel values remains the same in the two images ---- Brightness constancy constraint u =( u,v ): displacement Residual error  (displaced frame difference)

Robust Error Metrics Squared Difference is senstive to outliers

Robust Error Metrics Robust norm of error ( Huber 1981; Hampel , Ronchetti , Rousseeuw et al. 1986; Black and Anandan 1996; Stewart 1999) Sum of Absolute Difference (L1 norm) Grows less quickly than the quadratic penalty associated with least squares E SAD is NOT differentiable at the origin, not well suited to gradient descent approaches SRD: sum of robust difference SAD: sum of absolue difference

Robust Error Metric Smoothly varying function (Black and Rangarajan (1996) ) Quadratic for small values but grows more slowly away from the origin Geman –McClure function a : constant that can be thought of as an outlier threshold   MAD: median absolute deviation 

Spatially Varying Weights Pixels that may lie outside of the boundaries Partially or completely downweight the contribution of certain pixels  Erase moving object for background alignment Weighted (or Windowed) SSD function w and w 1 are zero outside the image boundaries

Weighted SSD If a large range of potential motions is allowed, the above metric can have a bias towards smaller overlap solutions. Answer: W indowed SSD score can be divided by the overlap area  E WSSD / A Overlap area S SD: S um of Squared Difference RMS: Root Mean Square Root Mean Square:

Bias and Gain (Exposure Differences) For images being aligned were not taken with the same exposure Simple model of linear intensity variation --- Bias and Gain model Gain Bias

Bias and Gain Least Squares with Bias and Gain Use linear regression to estimate gain and bias Color image Estimate bias and gain for each color channel Weighted prediction in video codecs

Correlation Cross-Correlation ( Confused with convolution ) Alternative to taking intensity difference Maximize the product of two aligned images

Correlation Cross-Correlation vs.  Convolution

Correlation Cross-Correlation Alternative to taking intensity difference Maximize the product of two aligned images If a very bright patch exists,  the maximum product may actually lie in that area.  

Normalized Cross-Correlation NCC in [-1,1] Works well when matching images taken with different exposures Performance  degrades for noisy low-contrast regions (Zero variance) Mean images of the corresponding patches 

Normalized Cross-Correlation Normalized SSD score ( Criminisi, Shotton, Blake et al., 2007). Produce comparable results to NCC S SD: S um of Squared Difference

How can I find in the image?

Idea #1: Template Matching Slow, combinatory, global solution

Idea #2: Pyramid Template Matching Faster, combinatory, locally optimal

Not guaranteed to produce the same results as a full search, but works almost as well and much faster Steps At coarser levels, search over a smaller number of discrete pixels Motion estimation at coarse level is used to initialize a smaller local search at the next finer level Hierarchical Motion Estimation

Hierarchical Motion Estimation Problem of image pyramid? N ot possible to coarsen the representation too much before significant features are blurred away.

Fourier-based Alignment 

Fourier-based Alignment 

Fourier-based Alignment  Fourier transform

Fourier-based Alignment 

Fourier-based Alignment 

Fourier transform of a shifted signal has the same magnitude as the original signal but a linearly varying phase Convolution in the spatial domain corresponds to multiplication in the Fourier domain  Fourier-based Alignment 

Fourier-based Alignment  Accelerated by Fast Fourier Transform (FFT) "Convolution in the spatial domain corresponds  to multiplication in the Fourier domain" Divide and Conquer​ O( N 2 ) -> O(N log N ) * See proof: https://www.youtube.com/watch?v=M0Sa8fLOajA  

Fourier-based Alignment  Consider the SSD formula. Its Fourier transform can be written as Thus, the SSD function can be computed by taking twice the correlation function and subtracting it from the sum of the energies in the two images.  S SD: S um of Squared Difference

Windowed Correlation  Convolution is the whole picture, but only a small part needs to be compared The cross-correlation function should be replaced with a windowed (weighted) cross-correlation function The weighting functions  ω and  ω 1 are zero outside the image boundaries

Phase Correlation  The spectrum of the two signals being matched is whitened by dividing each per-frequency product by the magnitudes of the Fourier transforms In the case of noiseless signals, we have I 1 ( x + u ) = I ( x )   The output of phase correlation is a single impulse located at the correct value of u . (Kuglin and Hines 1975; Brown 1992)

Rotations and Scale Pure rotation: Re-sample the images into polar coordinates Rotation and Scale: Re-sample the images into log-polar coordinates While Fourier-based alignment is mostly used to estimate translational shifts between images, it can also be used to estimate in-plane rotations and scales.

How can I find in the image?

Some notation before we get into the math… 2D image transformation 2D image coordinate Parameters of the transformation Translation transform coordinate Affine Pixel value at a coordinate affine transform coordinate Warped image can be written in matrix form when linear affine warp matrix can also be 3x3 when last row is [0 0 1]

Affine transform affine transform coordinate

Image alignment (problem definition) warped image template image Find the warp parameters p such that the SSD is minimized How could you find a solution to this problem?

Hard to optimize What can you do to make it easier to solve? assume good initialization, linearized objective and update incrementally This is a non-linear (quadratic) function of a non-parametric function! (Function I is non-parametric)

Idea #3: Model refinement Fastest, locally optimal (when you have a good initial solution)

Lucas- Kanade alignment A n I terative I mage R egistration T echnique with an A pplication to S tereo V ision ,  1981.

If you have a good initial guess p … can be written as … (a small incremental adjustment) (pretty strong assumption) (this is what we are solving for now)

How can we linearize the function I for a really small perturbation of p ? Taylor series approximation! This is still a non-linear (quadratic) function of a non-parametric function! (Function I is non-parametric)

Multivariable Taylor Series Expansion (First order approximation)

Multivariable Taylor Series Expansion (First order approximation) Recall: chain rule short-hand short-hand

Linear approximation Multivariable Taylor Series Expansion (First order approximation) What are the unknowns here?

The Jacobian (A matrix of partial derivatives)

The Jacobian Affine transform Rate of change of the warp (A matrix of partial derivatives)

Linear approximation Multivariable Taylor Series Expansion (First order approximation) Now, the function is a linear function of the unknowns

pixel coordinate (2 x 1) image intensity (scalar) warp function (2 x 1) warp parameters (6 for affine) image gradient (1 x 2) Partial derivatives of warp function (2 x 6) incremental warp (6 x 1) template image intensity (scalar)

Summary (of Lucas-Kanade Image Alignment) Solve for increment Taylor series approximation Linearize Difficult non-linear optimization problem Assume known approximate solution Strategy: Problem: then solve for warped image template image

OK, so how do we solve this?

Another way to look at it… (moving terms around) Have you seen this form of optimization problem before? vector of variables vector of constants constant

Another way to look at it… How do you solve this? Looks like variable constant constant

Least squares approximation is solved by is optimized when where Applied to our tasks: after applying H :  Hessian matrix

The Hessian (A square matrix of second-order partial derivatives) Jacobian of ▽f

Solve for increment Taylor series approximation Linearize Difficult non-linear optimization problem Assume known approximate solution Strategy: Solve: warped image template image Solution: Solution to least squares approximation Hessian

Gauss-Newton gradient descent   non-linear optimization! This is called…

Warp image Compute error image Compute gradient Evaluate Jacobian Compute Hessian Compute Update parameters Lucas Kanade (Additive alignment) x ’coordinates of the warped image (gradients of the warped image)

Compositional alignment Creating Full View Panoramic Image Mosaics and Environment Maps ,  R. Szeliski, H. Y. Shum 1997.

Image Alignment (start with an initial solution, match the image and template)

Additive Alignment incremental perturbation of parameters Image Alignment Objective Function Given an initial solution…several possible formulations

Compositional Alignment incremental warps of image Image Alignment Objective Function Given an initial solution…several possible formulations Additive Alignment incremental perturbation of parameters

Additive strategy go back, adjust and try again first shot second shot

Compositional strategy start from here first shot second shot

W(x;0 + D p) = W(x; D p) T (x) W(x;p) W(x;p) o W(x; D p) Compositional I (x) T (x) Additive I (x) W( x;p ) W(x;p+ D p)

Compositional Alignment Assuming an initial solution p and a compositional warp increment Original objective function (SSD) Another way to write the composition Skipping over the derivation…the new update rule is Identity warp

So what’s so great about this compositional form?

The Jacobian is constant. Jacobian can be precomputed! linearized form linearized form Jacobian of W(x;p) Jacobian of W(x;0) Additive Alignment Compositional Alignment

Compositional Image Alignment Minimize W(x;0 + D p) = W(x; D p) T (x) W(x;p) W(x;p) o W(x; D p) Jacobian is simple and can be precomputed

Warp image Compute error image Compute gradient Evaluate Jacobian Compute Hessian Compute Update parameters Lucas Kanade (Additive alignment)

Warp image Compute error image Compute gradient Evaluate Jacobian Compute Hessian Compute Update parameters Shum-Szeliski (Compositional alignment)

Any other speed up techniques?

Inverse alignment

Baker-Matthews alignment Lucas- kanade 20 years on: A unifying framework,  2004.

Why not compute warp updates on the template? Additive Alignment Compositional Alignment What happens if you let the template be warped too? Inverse Compositional Alignment

W(x;0 + D p) = W(x; D p) T (x) W(x;p) W( x;p ) o W( x; D p ) T (x) W(x;0 + D p) = W(x; D p) W(x;p) Compositional Inverse compositional W( x;p ) o W( x; D p ) -1 I (x) I (x)

Compositional strategy start from here first shot second shot

Inverse Compositional strategy first shot move the hole

So what’s so great about this inverse compositional form?

Inverse Compositional Alignment Minimize Solution Update can be precomputed from template!

Properties of inverse compositional alignment Jacobian can be precomputed It is constant - evaluated at W(x;0) Gradient of template can be precomputed It is constant Hessian can be precomputed Warp must be invertible (main term that needs to be computed)

Warp image Compute error image Compute gradient Evaluate Jacobian Compute Hessian Compute Update parameters Lucas Kanade (Additive alignment)

Warp image Compute error image Compute gradient Evaluate Jacobian Compute Hessian Compute Update parameters Shum-Szeliski (Compositional alignment)

Warp image Compute error image Compute gradient Evaluate Jacobian Compute Hessian Compute Update parameters Baker-Matthews (Inverse Compositional alignment)

Algorithm Efficient Authors Forwards Additive No Lucas, Kanade Forwards compositional No Shum, Szeliski Inverse Additive Yes Hager, Belhumeur Inverse Compositional Yes Baker, Matthews

An Iterative Image Registration Technique with an Application to Stereo Vision. 1981 Lucas Kanade Detection and Tracking of Feature Points. 1991 Kanade Tomasi Good Features to Track. 1994 Tomasi Shi History of the Kanade-Lucas-Tomasi (KLT) Tracker The original KLT algorithm

Method for aligning (tracking) an image patch Kanade-Lucas-Tomasi Method for choosing the best feature (image patch) for tracking Lucas- Kanade Tomasi-Kanade How should we select features? How should we track them from frame to frame?

9.2 Optical flow

optical flow

optical flow used for motion estimation in visual odometry

optical flow used for segments the background

Image Alignment Lucas-Kanade (Forward additive) Baker-Matthews (Inverse Compositional)

Tracking in Video KLT Mean shift Kalman Filtering SLAM

Optical Flow

Optical Flow Problem Definition Assumptions Brightness constancy Small motion Given two consecutive image frames, estimate the motion of each pixel

Optical Flow (Problem definition) Estimate the motion (flow) between these two consecutive images How is this different from estimating a 2D transform?

Key Assumptions (unique to optical flow) Color Constancy (Brightness constancy for intensity images) Small Motion (pixels only move a little bit) Implication: allows for pixel to pixel comparison (not image features) Implication: linearization of the brightness constancy constraint

Approach Look for nearby pixels with the same color (small motion) (color constancy)

Brightness constancy Assumption:Brightness of the point will remain the same constant Scene point moving through image sequence Assumption 1

Small motion Optical flow (velocities): Displacement: … the brightness between two consecutive image frames is the same For a really small space-time step … Assumption 2

total derivative partial derivative Equa tion is not obvious. Where does this come from? These assumptions yield the … Brightness Constancy Equation

For small space-time step, brightness of a point is the same Insight: If the time step is really small, we can linearize the intensity function

Multivariable Taylor Series Expansion (First order approximation, two variables)

assuming small motion divide by take limit Brightness Constancy Equation Multivariable Taylor Series Expansion (First order approximation, two variables)

shorthand notation vector form Brightness Constancy Equation (x-flow) (y-flow) (1 x 2) (2 x 1)

(putting the math aside for a second…) What do the term of the brightness constancy equation represent? flow velocities temporal gradient How do you compute these terms? Image gradients (at a point p)

spatial derivative temporal derivative Forward difference Sobel filter Derivative-of-Gaussian filter … How do you compute … frame differencing

Frame differencing 1 1 1 1 1 1 1 1 1 1 1 10 10 10 10 1 10 10 10 10 1 10 10 10 10 1 10 10 10 10 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 10 10 10 1 1 10 10 10 1 1 10 10 10 9 9 9 9 9 9 9 (example of a forward difference) - =

Example: 1 1 1 1 1 1 1 1 1 1 1 10 10 10 10 1 10 10 10 10 1 10 10 10 10 1 10 10 10 10 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 10 10 10 1 1 10 10 10 1 1 10 10 10 y x y x - - - - - 9 - - 9 - - 9 - - 9 - y x - - - - - 9 9 9 9 - - - - - - y x 9 9 9 9 9 9 9 y x 3 x 3 patch -1 0 1 -1 1

spatial derivative optical flow Forward difference Sobel filter Derivative-of-Gaussian filter … How do you compute … temporal derivative frame differencing How do you compute this? We need to solve for this! (this is the unknown in the optical flow problem)

Solution lies on a straight line The solution cannot be determined uniquely with a single constraint (a single pixel) many combinations of u and v will satisfy the equality

known unknown

Horn- Schunck Optical Flow (1981) Lucas- Kanade Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow (flow can vary from pixel to pixel) brightness constancy method of differences global method (dense) local method (sparse) small motion

Constant flow

Where do we get more equations (constraints)? Assume that the surrounding patch (say 5x5) has ‘ constant flow’

Flow is locally smooth Neighboring pixels have same displacement Using a 5 x 5 image patch, gives us 25 equations Assumptions:

Equivalent to solv ing : where the summation is over each pixel p in patch P

where the summation is over each pixel p in patch P Sometimes called ‘Lucas- Kanade Optical Flow’ (can be interpreted to be a special case of the LK method with a translational warp model) Equivalent to solv ing :

OpenCV Lucas- Kanade

When is this solvable?

When is this solvable? A T A should be invertible A T A should not be too small A T A should be well conditioned l 1 and l 2 should not be too small l 1 / l 2 should not be too large ( l 1 =larger eigenvalue)

Where have you seen this before? Harris Corner Detector! = What are the implications ?

Implications Corners are when λ1, λ2 are big; this is also when Lucas-Kanade optical flow works best Corners are regions with two different directions of gradient (at least) Corners are good places to compute flow! What happens when you have no ‘corners’?

Barber’s pole illusion

Barber’s pole illusion

Barber’s pole illusion

Aperture Problem In which direction is the line moving? small visible image patch

Aperture Problem In which direction is the line moving? small visible image patch

Aperture Problem

Aperture Problem

Aperture Problem

Aperture Problem

Want patches with different gradients to the avoid aperture problem

Want patches with different gradients to avoid aperture problem

1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 5 5 5 5 - - - - - 1 1 1 - 2 2 2 - 3 3 3 - 4 4 4 y x y x optical flow: (1,1) H(x,y) = y I(x,y) We recover the v of the optical flow but not the u. This is the aperture problem. Compute gradients Solution:

Horn- Schunck optical flow

Horn- Schunck Optical Flow (1981) Lucas- Kanade Optical Flow (1981) ‘constant’ flow (flow is constant for all pixels) ‘smooth’ flow (flow can vary from pixel to pixel) brightness constancy method of differences global method (dense) local method (sparse) small motion

Smoothness most objects in the world are rigid or deform elastically moving together coherently we expect optical flow fields to be smooth

Key idea (of Horn-Schunck optical flow) Enforce brightness constancy Enforce smooth flow field to compute optical flow

Key idea (of Horn-Schunck optical flow) Enforce brightness constancy Enforce smooth flow field to compute optical flow

For every pixel, lazy notation for Enforce brightness constancy

Key idea (of Horn-Schunck optical flow) Enforce brightness constancy Enforce smooth flow field to compute optical flow

Enforce smooth flow field u-component of flow

Which flow field optimizes the objective? ?

small big Which flow field optimizes the objective?

Key idea (of Horn-Schunck optical flow) Enforce brightness constancy Enforce smooth flow field to compute optical flow bringing it all together…

Horn-Schunck optical flow smoothness brightness constancy weight

HS optical flow objective function Brightness constancy Smoothness

How do we solve this minimization problem?

Compute partial derivative, derive update equations (gradient de s cent!) How do we solve this minimization problem?

Compute the partial derivatives of this huge sum! smoothness term brightness constancy

Compute the partial derivatives of this huge sum! it’s not so bad… how many u terms depend on k and l? ONE from brightness constancy FOUR from smoothness

Compute the partial derivatives of this huge sum! Smoothness - Square of the magnitude of the gradient - Sum of the squares of the Laplacians Approximations of the Laplacian:  

Compute the partial derivatives of this huge sum! (variable will appear four times in sum)

Compute the partial derivatives of this huge sum! short hand for local average (variable will appear four times in sum)

Where are the extrema of E? (set derivatives to zero and solve for unknowns u and v)

Where are the extrema of E? (set derivatives to zero and solve for unknowns u and v) how do you solve this? this is a linear system

ok, take a step back, why are we doing all this math?

We are solving for the optical flow (u,v) given two constraints smoothness brightness constancy We need the math to minimize this (back to the math)

Where are the extrema of E? (set derivatives to zero and solve for unknowns u and v) how do you solve this? Partial derivatives of Horn-Schunck objective function E:

Recall

Recall Adjugate matrix : = Determinant :

Recall (det A) Same as the linear system: (det A) (d et A) =

Rearrange to get update equations: new value old average

new value old average When lambda is small (lambda inverse is big)… Recall:

new value old average When lambda is small (lambda inverse is big)… goes to zero goes to zero …we only care about smoothness. Recall:

We are solving for the optical flow (u,v) given two constraints smoothness brightness constancy We needed the math to minimize this (now to the algorithm)

Horn-Schunck Optical Flow Algorithm Precompute image gradients Precompute temporal gradients Initialize flow field While not converged Compute flow field updates for each pixel:

HW3 due June 8, 2021 Implementing Horn &  Schunck  optical flow estimation as above​ Synthetically translate lena.bmp one pixel to the right and downward​ Try

HW3 Precompute image gradients Precompute temporal gradients

HW3 1 pixel

HW3

HW3

HW3 3.   Initialize flow field 4. While not converged   Compute flow field updates for each pixel: 32 iterations 64 iterations

HW3

HW3 Iteration = 64 Iteration = 16 Iteration = 4 Iteration = 1 Scale = 10

OpenCV Optical Flow Algorithm https://docs.opencv.org/3.4/d4/dee/tutorial_optical_flow.html

Rolling shutter wobble removal

Rolling shutter wobble removal To compensate for these distortion, which are caused by different exposure times for different scanlines, accurate per-pixel optic flow must be estimated. (Section 9.2.1) Modeling rolling shutter is also important to obtain accurate pose estimates in structure from motion

Deep Learning https://www.youtube.com/watch?v=OSEuYBwOSGI ECCV 2020 Best Paper Award | RAFT: A New Deep Network Architecture For Optical Flow |

Deep Learning https://www.youtube.com/watch?v=OSEuYBwOSGI ECCV 2020 Best Paper Award | RAFT: A New Deep Network Architecture For Optical Flow |

9.3 Parametric motion

Spline-based Motion Traditionally, optical flow algorithms compute an independent motion estimate for each pixel. Alternative approach is to represent the motion field as a two-dimensional spline controlled by a smaller number of control vertices Traditionally, optical flow algorithms compute an independent motion estimate for each pixel. Alternative approach is to represent the motion field as a two-dimensional spline controlled by a smaller number of control vertices Spline-Based Image Registration, R. Szeliski , J. Coughlan, 1997

control vertices basis functions  Two-dimensional spline controlled by a smaller number of control vertices ​

control vertices basis functions

control vertices basis functions

Undistorted Barrel distortion Pincushion distorti on

https://www.youtube.com/watch?v=yAB6OQsHOyI

9.4 Layered Motion

Layered Motion In many situations, visual motion is caused by the movement of a small number of objects at different depths in the scene.

Layered Motion In many situations, visual motion is caused by the movement of a small number of objects at different depths in the scene.

Layered Motion In many situations, visual motion is caused by the movement of a small number of objects at different depths in the scene.

Layered Motion In many situations, visual motion is caused by the movement of a small number of objects at different depths in the scene.

Layered motion representations not only lead to compact   representations but they also exploit the information available   in multiple video frames, as well as accurately modeling the   appearance of pixels near motion discontinuities.  

Layered Motion How to compute layered representation of a video?  Estimate affine motion models over a collection of non-overlapping patches  Cluster the estimates using K-means  Alternate between assigning pixels to layers and recomputing motion estimates for each layer using the assigned pixels Layers are constructed by warping and merging the various layer   pieces from all of the frames together.   Wang and Adelson (1994)

Layered Motion Baker, Szeliski, and Anandan (1998)

Transparent Layers and Reflections Reflection in windows, picture frames, … Reflection Model how much intensity each layer contributed to the final image Glass surface Image transmission reflection

11 Reflection Transparent motion separation The amount of reflected light is quite low compared to the transmitted light (the picture of the girl) and yet the algorithm is still able to recover both layers.

Video object tracking One of the most widely used applications of computer vision to video analysis is video object tracking https://www.youtube.com/watch?v=LjjJQ81RbX0

10 Questions What is the assumption of the Lucas- Kanade alignment method? P47 Tell the flowchat of Lucas Kanade alignment (Additive alignment)? P64 What’s different between Additive alignment and inverse compositional alignment? P87 What are the assumptions of the Optical flow? P101 What happens when you have no ‘corners’ in optical flow? p130-140

10 Questions What’s the topic of HW3? P142 What’s different between Horn- Schunck optical flow and Lucas- Kanade optical flow? P143 What are two key ideas for Horn- Schunck optical flow? P146 When is the deadline for hw3? P175 What are the applications of Layer motion (Give 3 examples)? P195 - 205

10 Answers If you have a good initial guess Ans : Warp image Compute error image Compute gradient Evaluate Jacobian Compute Hessian Compute Update parameters Ans : Jacobian can be precomputed Gradient of template can be precomputed Hessian can be precomputed Warp must be invertible

10 Answers Ans : Brightness constancy Small motion Ans : Barber’s pole illusion Aperture Problem Horn- Schunck optical flow Smooth flow vs Constant flow

10 Answers Ans : Brightness constancy Smooth flow field June 8, 2021 Ans : segments the background Transparent Layers and Reflections Video object tracking Video object segmentation
Tags