ImageTracking.Farneback — TypeFarneback(Args...)A method for dense optical flow estimation developed by Gunnar Farneback. It computes the optical flow for all the points in the frame using the polynomial representation of the images. The idea of polynomial expansion is to approximate the neighbourhood of a point in a 2D function with a polynomial. Displacement fields are estimated from the polynomial coefficients depending on how the polynomial transforms under translation.
Options
Various options for the fields of this type are described in more detail below.
Choices for iterations
Number of iterations the displacement estimation algorithm is run at each point. If left unspecified a default value of seven iterations is assumed.
Choices for estimation_window
Determines the neighbourhood size over which information will be intergrated when determining the displacement of a pixel. The total size equals 2*estimation_window + 1. If left unspecified a default value of eleven is assumed.
Choices for σ_estimation_window
Standard deviation of a Gaussian weighting filter used to weigh the contribution of a pixel's neighbourhood when determining the displacement of a pixel. If left unspecified a default value of nine is assumed.
Choices for expansion_window
Determines the size of the pixel neighbourhood used to find polynomial expansion for each pixel; larger values mean that the image will be approximated with smoother surfaces, yielding more robust algorithm and more blurred motion field. The total size equals 2*expansion_window + 1. If left unspecified a default value of six is assumed.
Choices for σ_expansion_window
Standard deviation of the Gaussian that is used to smooth the image for the purpose of approximating it with a polynomial expansion. If left unspecified a default value of five is assumed.
References
- Farnebäck G. (2003) Two-Frame Motion Estimation Based on Polynomial Expansion. In: Bigun J., Gustavsson T. (eds) Image Analysis. SCIA 2003. Lecture Notes in Computer Science, vol 2749. Springer, Berlin, Heidelberg
- Farnebäck, G.: Polynomial Expansion for Orientation and Motion Estimation. PhD thesis, Linköping University, Sweden, SE-581 83 Linköping, Sweden (2002) Dissertation No 790, ISBN 91-7373-475-6.
ImageTracking.LucasKanade — TypeLucasKanade(Args...)A differential method for optical flow estimation developed by Bruce D. Lucas and Takeo Kanade. It assumes that the flow is essentially constant in a local neighbourhood of the pixel under consideration, and solves the basic optical flow equations for all the pixels in that neighbourhood by the least squares criterion.
Options
Various options for the fields of this type are described in more detail below.
Choices for iterations
The termination criteria of the iterative search algorithm, that is, the number of iterations.
Choices for window_size
Size of the search window at each pyramid level; the total size of the window used is 2*window_size + 1. If left unspecified, a default value of eleven is assumed.
Choices for pyramid_levels
0-based maximal pyramid level number; if set to 0, pyramids are not used (single level), if set to 1, two levels are used, and so on. IF left unspecified, a default value of four is assumed.
Choices for eigenvalue_threshold
The algorithm calculates the minimum eigenvalue of a (2 x 2) normal matrix of optical flow equations, divided by number of pixels in a window; if this value is less than eigenvalue_threshold, then a corresponding feature is filtered out and its flow is not processed (default value is 1e-4).
ϵ termination criteria
Minimum required change in displacement at which the iterative algorithm continues to work. Default value is 1e-2.
References
- B. D. Lucas, & Kanade. "An Interative Image Registration Technique with an Application to Stereo Vision," DARPA Image Understanding Workshop, pp 121-130, 1981.
- J.-Y. Bouguet, “Pyramidal implementation of the affine lucas-kanade feature tracker description of the algorithm,” Intel Corporation, vol. 5,no. 1-10, p. 4, 2001.
ImageTracking.OpticalFlowAlgorithm — TypeOpticalFlowAlgorithmAn optical flow algorithm with given parameters.
ImageTracking.calculate_statistics — Methodcalculate_statistics(error)Quantifies the flow error in terms of a mean and standard deviation as well as two additional statistics that measure "robustness".
Details
RX and AX are the robustness statistics. RX denotes the percentage of pixels that have an error measure over X. AX denotes the accuracy of the error measure at the Xth percentile.
Arguements
The error parameters needs to be two-dimensional arrays of length-2 vectors (of type SVector) which represent the error between two flows.
Example
Compute the Mean, SD, RX, AX stats of the flow error calculated using endpoint error method.
using ImageTracking
error = evaluate_flow_error(ground_truth_flow, estimated_flow, EndpointError())
mean, sd, rx, ax = calculate_statistics(error)Compute the mean, SD, RX, AX stats of the flow error calculated using angula error method.
using ImageTracking
error = evaluate_flow_error(ground_truth_flow, estimated_flow, AngularError())
mean, sd, rx, ax = calculate_statistics(error)References
[1] S. Baker, D. Scharstein, JP Lewis, S. Roth, M.J. Black, and R. Szeliski. A database and evaluation methodology for optical flow. International Journal of Computer Vision, 92(1):1–31, 2011.
ImageTracking.evaluate_flow_error — Methodevaluate_flow_error(ground_truth_flow, estimated_flow, AngularError())Returns a 2-Dimensional array that matches the dimensions of the input flow vector and that depicts the angle error between the estimated flow and the ground truth flow.
Details
If the estimated flow at a point is (u0, v0) and ground truth flow is (u1, v1), it calculates the angle between (u0, v0, 1) and (u1, v1, 1) vectors as measure for error.
Arguments
The flow parameters needs to be two-dimensional arrays of length-2 vectors (of type SVector) which represent the displacement of each pixel.
Example
Compute the angle error between two flows.
using ImageTracking
result = evaluate_flow_error(ground_truth_flow, estimated_flow, AngularError())
imshow(result)References
[1] S. Baker, D. Scharstein, JP Lewis, S. Roth, M.J. Black, and R. Szeliski. A database and evaluation methodology for optical flow. International Journal of Computer Vision, 92(1):1–31, 2011.
ImageTracking.evaluate_flow_error — Methodevaluate_flow_error(ground_truth_flow, estimated_flow, EndpointError())Returns a 2-Dimensional array that matches the dimensions of the input flow vector and that depicts the end point error between the estimated flow and the ground truth flow.
Details
If the estimated flow at a point is (u0, v0) and ground truth flow is (u1, v1), then error will be sqrt[(u0 - u1)^2 + (v0 - v1)^2] at that point.
Arguments
The flow parameters needs to be two-dimensional arrays of length-2 vectors (of type SVector) which represent the displacement of each pixel.
Example
Compute the end point error between two flows.
using ImageTracking
result = evaluate_flow_error(ground_truth_flow, estimated_flow, EndpointError())
imshow(result)References
[1] S. Baker, D. Scharstein, JP Lewis, S. Roth, M.J. Black, and R. Szeliski. A database and evaluation methodology for optical flow. International Journal of Computer Vision, 92(1):1–31, 2011.
ImageTracking.get_pyramid_coordinate — MethodArguments
level: Level of the pyramid in[1, levels]range.
ImageTracking.haar_coordinates — Methodcoordinates = haar_coordinates(height, width, feat)Returns an array containing the coordinates of the Haar-like features of the specified type.
The caller of the haar_coordinates function specifies a rectangular region in the image by its height and width and a particular feature type, e.g. two rectangles side by side (:x2). This function then computes the coordinates for the particular rectangle configuration for all possible locations, and all possible rectangle widths & heights.
Parameters:
- height = Height of the neighbourhood/window where the coordinates are computed
- width = Width of the neighbourhood/window where the coordinates are computed
- feat = A symbol specifying the type of the Haar-like feature to be found
Currently, feat can take 5 values:
:x2 = Two rectangles along horizontal axis
+---------------------------------------+
| |
| +-----+-----+ |
| | | | |
| | -1 | +1 | +----------> |
| | | | |
| +-----+-----+ |
| |
| + |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| v |
| |
| |
| |
| |
| |
| |
+---------------------------------------+
:y2 = Two rectangles along vertical axis
+---------------------------------------+
| |
| +------------+ |
| | -1 | |
| +------------+ +----------> |
| | +1 | |
| +------------+ |
| |
| + |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| v |
| |
| |
| |
| |
| |
| |
+---------------------------------------+
:x3 = Three rectangles along horizontal axis
+---------------------------------------+
| |
| +-----+-----+-----+ |
| | | | | |
| | -1 | +1 | -1 | +----------> |
| | | | | |
| +-----+-----+-----+ |
| |
| + |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| v |
| |
| |
| |
| |
| |
| |
+---------------------------------------+
:y3 = Three rectangles along vertical axis
+---------------------------------------+
| |
| +------------+ |
| | -1 | |
| +------------+ |
| | +1 | +----------> |
| +------------+ |
| | -1 | |
| +------------+ |
| |
| + |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| v |
| |
| |
| |
| |
+---------------------------------------+
:xy4 = Four rectangles along horizontal and vertical axes
+---------------------------------------+
| |
| +-----+-----+ |
| | | | |
| | -1 | +1 | |
| | | | |
| +-----+-----+ +----------> |
| | | | |
| | +1 | -1 | |
| | | | |
| +-----+-----+ |
| |
| + |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| v |
| |
| |
+---------------------------------------+
The +1 and -1 signs show which rectangles are subtracted and which are added to evaluate the final haar feature.
ImageTracking.haar_features — Methodfeatures = haar_features(img, top_left, bottom_right, feat)
features = haar_features(img, top_left, bottom_right, feat, coordinates)Returns an array containing the Haar-like features for the given Integral Image in the region specified by the points topleft and bottomright.
The caller of the haarfeatures function specifies a rectangular region in the image by its topleft and bottom_right points and a particular feature type, e.g. two rectangles side by side (:x2). This function then computes the values of all haar rectangular features for the particular rectangle configuration for all possible locations, and all possible rectangle widths & heights if coordinates of certain specific features are not provided. If they are provided then it only computes the values of those features whose coordinates are given. Calculating value of haar rectangular feature corresponds to finding the difference of sums of all points in the different rectangles comprising the feature.
Parameters:
- img = The Integral Image for which the Haar-like features are to be found
- top_left = The top and left most point of the region where the features are to be found
- bottom_right = The bottom and right most point of the region where the features are to be found
- feat = A symbol specifying the type of the Haar-like feature to be found
Currently, feat can take 5 values:
:x2 = Two rectangles along horizontal axis
+---------------------------------------+
| |
| +-----+-----+ |
| | | | |
| | -1 | +1 | +----------> |
| | | | |
| +-----+-----+ |
| |
| + |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| v |
| |
| |
| |
| |
| |
| |
+---------------------------------------+
:y2 = Two rectangles along vertical axis
+---------------------------------------+
| |
| +------------+ |
| | -1 | |
| +------------+ +----------> |
| | +1 | |
| +------------+ |
| |
| + |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| v |
| |
| |
| |
| |
| |
| |
+---------------------------------------+
:x3 = Three rectangles along horizontal axis
+---------------------------------------+
| |
| +-----+-----+-----+ |
| | | | | |
| | -1 | +1 | -1 | +----------> |
| | | | | |
| +-----+-----+-----+ |
| |
| + |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| v |
| |
| |
| |
| |
| |
| |
+---------------------------------------+
:y3 = Three rectangles along vertical axis
+---------------------------------------+
| |
| +------------+ |
| | -1 | |
| +------------+ |
| | +1 | +----------> |
| +------------+ |
| | -1 | |
| +------------+ |
| |
| + |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| v |
| |
| |
| |
| |
+---------------------------------------+
:xy4 = Four rectangles along horizontal and vertical axes
+---------------------------------------+
| |
| +-----+-----+ |
| | | | |
| | -1 | +1 | |
| | | | |
| +-----+-----+ +----------> |
| | | | |
| | +1 | -1 | |
| | | | |
| +-----+-----+ |
| |
| + |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| v |
| |
| |
+---------------------------------------+
The +1 and -1 signs show which rectangles are subtracted and which are added to evaluate the final haar feature.
- coordinates = The user can provide the coordinates of the rectangles if only certain Haar-like features are to found in the given region. The required format is a 3 Dimensional array as (f,r,4) where f = numberoffeatures, r = numberofrectangles and 4 is for the coordinates of the topleft and bottomright point of the rectangle (toplefty, topleftx, bottomrighty, bottomrightx). The default value is nothing, where all the features are found
References
M. Oren, C. Papageorgiou, P. Sinha, E. Osuna and T. Poggio, "Pedestrian detection using wavelet templates," Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, 1997, pp. 193-199.
P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features," Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, 2001, pp. I-511-I-518 vol.1.
ImageTracking.optical_flow — Methodflow = optical_flow(source, target, Farneback(Args...))
flow = optical_flow(source, target, displacement, Farneback(Args...))Returns the dense optical flow from the source to the target image using the Farneback algorithm.
Details
The source and target images must be Gray types.
The displacement argument allows you to specify an initial guess for the optical flow and must be of type Array{SVector{2, Float64}, 2}. The elements of displacement should represent the flow required to map the (row, column) of each pixel in the source image to the target image.
ImageTracking.optical_flow — Methodflow, indicator = optical_flow(source, target, points, LucasKanade(Args...))
flow, indicator = optical_flow(source, target, points, displacement, LucasKanade(Args...))Returns the optical flow from the source to the target image for the specified points using the LucasKanade algorithm.
Details
The source and target images must be Gray types.
The points argument is of type Array{SVector{2, Float64}, 2} and represents a set of keypoints in the source image for which the optical flow is to be computed. The coordinates of a point represent the (row, column) of a pixel in the image. The displacement argument allows you to specify an initial guess for the optical flow.
The function returns flow of type Array{SVector{2, Float64}, 2} which matches the length of points and represents the displacement needed to map a point in the source image to the target image.
The indicator is a vector of boolean values (one for each point in points) which signals whether a particular point was successfully tracked or not.
In order to use the flow to index the corresponding point in the target image you first need to round it to the nearest integer, and check that it falls within the bounds of the target image dimensions.
ImageTracking.polynomial_expansion — MethodA, B, C = polynomial_expansion(implementation, img, neighbourhood, σ)Returns the polynomial coefficients of the approximation of the neighbourhood of a point in a 2D function with a polynomial. The expansion is: f(x) = x'Ax + B'x + C
Options
Various options for the fields of this type are described in more detail below.
Choices for implementation
Selects implementation (::MatrixImplementation vs ::ConvolutionImplementation). The matrix implementation serves as a useful reference against which other faster implementations can be validated.
Choices for img
Grayscale (Float) image whose polynomial expansion is to be found.
Choices for window_size
Determines the size of the pixel neighbourhood used to find polynomial expansion for each pixel; larger values mean that the image will be approximated with smoother surfaces, yielding more robust algorithm and more blurred motion field. The total size equals 2*window_size + 1.
Choices for σ
Standard deviation of the Gaussian that is used to smooth the image for the purpose of approximating it with a polynomial expansion.
References
Farnebäck, G.: Polynomial Expansion for Orientation and Motion Estimation. PhD thesis, Linköping University, Sweden, SE-581 83 Linköping, Sweden (2002) Dissertation No 790, ISBN 91-7373-475-6.
ImageTracking.visualize_flow — Methodvisualize_flow(flow, ColorBased(), RasterConvention())
visualize_flow(flow, ColorBased(), CartesianConvention())Returns an image that matches the dimensions of the input matrix and that depicts the orientation and magnitude of the optical flow vectors using the HSV color space.
Details
Hue encodes angle between optical flow vector and the x-axis in the image plane; saturation encodes the ratio between the individual vector magnitudes and the maximum magnitude among the whole motion field, and the values always equal one.
Arguments
The flow parameter needs to be a two-dimensional arrays of length-2 vectors (of type SVector) which represent the displacement of each pixel. The displacement can be expressed in a coordinate system based on a RasterConvention() (i.e. rows and columns of a matrix) or a CartesianConvention().
Example
Compute the HSV encoded visualization of flow vectors in (row, column) convention.
using ImageTracking
hsv = visualize_flow(flow, ColorBased(), RasterConvention())
imshow(RGB.(hsv))Compute the HSV encoded visualization of flow vectors in (x, y) convention.
using ImageTracking
hsv = visualize_flow(flow, ColorBased(), CartesianConvention())
imshow(RGB.(hsv))References
[1] S. Baker, D. Scharstein, JP Lewis, S. Roth, M.J. Black, and R. Szeliski. A database and evaluation methodology for optical flow. International Journal of Computer Vision, 92(1):1–31, 2011.