Dense Motion Estimation for Computer Vision

Computer Vision

1. Dense Motion Estimation Kemal Alperen Çetiner 504221573

2. TABLE OF CONTENTS 01 Introduction What is motion estimation? 02 Metrics How to derive the metrics for the problem 03 Optimization How to find the optimal variables 04 Optical Flow General case of motion estimation 05 Layered Motion Multilayer motion estimation 06 Application Rolling shutter problem

3. What is motion estimation? How is camera moving? How many moving objects? What is the direction of moving objects? How fast the objects are moving?

4. Application Examples Background Subtraction Stationary Camera Static Background Video Stabilization Moving Camera

5. How to approach the task? Define the metrics Adhering to assumptions Robustness Complying with techniques Device the techniques Time-Computation complexity Accuracy Adapt for the conditions Methods for specific conditions

6. Metrics : First Assumption Residual Error Sum of squared differences Template Image Camera Image Brightness Constancy Constraint Considerations: Subpixels? Colors?

7. Way to robustness : Bias & Gain Different exposure for different frames Bias and gain may differ for color channels

8. Way to robustness : outliers & weights More robust to outliers in comparison to L2 metric Geman–McClure function Pixels may remain outside the image Weights/Windows to eliminate the problem Normalization to prevent size differences

9. Way to robustness: Correlation Problem of very bright areas Solving the gain & bias problem by cross-correlation Normalized Cross-Correlation Problem: Noisy low contrast regions

10. How to optimize the Metric : Full Search Full search over block of pixels Coarse-to-Fine approach to accelerate the computation Warp or change the initialization

11. How to optimize the Metric : Fourier Same magnitude varying phase Convolution-Fourier Transform relation with cross-correlation

12. How to optimize the Metric : Fourier Whitened Correlation Phase signal is an impulse peaking at shift u Performance depends on the signal characteristic of original image Low SNR may result accentuation of noise Under high SNR it can provide better results than regular correlation

13. How to optimize the Rotation Pure Rotation FFT can used to estimate it similar to shift Rotation + Scaling + Small Translation Approach: Images are converted to the Fourier domain Magnitude images are aligned in rotation and scale using the polar or log-polar representations Rotation and scale are estimated One of the images can be de-rotated and scaled Regular translational algorithm is applied for translational shift

14. How to optimize the Incrementally Methods so far estimate the alignment to the nearest pixel given the search precision is one pixel Taylor series approximation Gradient descent on the SSD energy function

15. How to optimize : Gauss-Newton Gradient descent on the SSD energy function Hessian and gradient-weighted residual vector Normal equation for the Gauss-Newton process

16. How to optimize : Aperture Problem Textureless Regions Edges Only the normal component of the velocity (displacement) can be reliably recovered Rank deficient

17. How to optimize : Aperture Problem

18. General Case : Optical Flow Motion estimation is to compute an independent estimate of motion at each pixel, which is generally known as optical (or optic) flow. The problem is underconstrained: Patch based approach Adding smoothness term Markov random fields and search for global minimum Horn and Schunck model Smooth over all image Regularize by the locality constraint Formulated as an energy function

19. General Case : Optical Flow Energy Function Smoothness Term Brightness Constraint

20. General Case : Robust Optical Flow Harder optimization process No closed form solution Approach Gradient Descent Iteratively reweighted LS

21. Layered Motion Motion is caused by objects from differing depths Grouping can provide benefits Segmentation Tracking Exploits the information available in multiple video frames Solves the flow problem near discontinuities

22. Layered Motion Estimate affine motion models over a collection of non-overlapping patches Cluster these estimates using k-means Alternate between assigning pixels to layers and recomputing motion estimates for each layer using the assigned pixels

23. Application : Rolling Shutter Effect Shutter reading process Global vs Rolling Shutter Motion estimation to remove the rolling shutter distortion IMU assisted motion estimation with mixture models