Motion-vector based registration

VideoRegistration

Robust 2D point-set registration: estimation of a global transform (and residual local motion) from point correspondences, such as those derived from codec block motion vectors via openvideo(...; export_mvs = true), VideoIO.motion_vectors, and VideoIO.correspondences.

Codec motion vectors are compression decisions, not measured optical flow. This module therefore emphasises robust fitting (RANSAC + closed-form model fits) and quality metrics, so that a caller can decide when to trust the codec-derived estimate, when to use it only as an initialization for conventional registration, and when to fall back entirely.

Typical usage

using VideoIO
using VideoIO.VideoRegistration

video = openvideo("video.mp4", export_mvs = true)
sz = out_frame_size(video)  # (width, height)
for frame in video
    mvs = motion_vectors(video)
    gm = estimate_global_motion(mvs; model = :similarity, frame_size = sz)
    conf = classify_confidence(gm)
    if conf === :high || conf === :moderate
        # use `gm.A` (a 2×3 matrix mapping current-frame points to
        # reference-frame points) directly, or as an initialization
        dx, dy = translation(gm)
    else
        # fall back to full image-domain registration
    end
end

(The estimate_global_motion(mvs; ...) motion-vector convenience method is VideoIO-side glue — see src/registration_glue.jl; the module itself only takes point matrices.)

Conventions

A fitted transform is stored as a 2×3 matrix A such that

[x_src, y_src] ≈ A * [x_dst, y_dst, 1]

i.e. it maps a point in the current frame (dst) to the corresponding point in the reference frame (src), matching FFmpeg's motion vector convention src = dst + motion / motion_scale. Use invert_transform for the reference→current direction.

Point sets are n×2 matrices with columns (x, y) in pixels.

estimate_global_motion(dst::AbstractMatrix, src::AbstractMatrix; kwargs...)

Estimate the global transform mapping the n×2 point set dst to src (e.g. current-frame → reference-frame correspondences from VideoIO.correspondences).

Returns a GlobalMotion, or nothing when there are fewer than min_points correspondences (e.g. I-frames) or RANSAC fails — callers should treat nothing as "no estimate available" and fall back to conventional registration.

Keyword arguments

• model = :similarity: :translation, :similarity, or :affine
• min_points = 8: minimum correspondences required
• threshold = 2.0, max_iterations = 2000, confidence = 0.99, rng: RANSAC parameters, see ransac_fit
• frame_size = nothing: (width, height); enables the coverage quality metric

VideoRegistration.estimate_global_motion(mvs::AbstractVector{MotionVector}; kwargs...)

Estimate the global frame-to-reference transform directly from decoder motion vectors: converts them to filtered point correspondences with correspondences and fits with VideoRegistration.estimate_global_motion(dst, src; ...).

Accepts the filter keyword arguments of correspondences (past_only, min_block_size, max_displacement, border, and frame_size, which also enables the coverage quality metric) in addition to the fitting keyword arguments of the point-matrix method.

GlobalMotion

Result of estimate_global_motion.

Fields

• model::Symbol: the fitted model (:translation, :similarity, :affine)
• A::Matrix{Float64}: 2×3 transform mapping current-frame (dst) points to reference-frame (src) points
• npoints::Int: number of correspondences after filtering
• inliers::BitVector: RANSAC inlier mask over those correspondences
• inlier_fraction::Float64
• median_residual::Float64: median reprojection error of the inliers, pixels
• coverage::Float64: fraction of the frame area spanned by the inlier points' bounding box (NaN if frame_size was not provided)

translation(gm::GlobalMotion) -> (tx, ty)

Translation component of the fitted transform.

rotation(gm::GlobalMotion) -> θ

In-plane rotation angle (radians) of the fitted transform. Meaningful for :similarity fits (and :affine fits with little shear).

scale_factor(gm::GlobalMotion) -> s

Uniform scale of the fitted transform. Meaningful for :similarity fits.

classify_confidence(gm; min_points = 30, min_inlier_fraction = 0.5,
                    max_median_residual = 1.0, min_coverage = 0.25)
    -> :high | :moderate | :low | :none

Classify a GlobalMotion estimate for decision making:

:high requires all thresholds to pass (coverage is only checked when it was computed, i.e. frame_size was provided); :moderate requires the inlier-fraction and residual checks; anything else is :low.

local_residuals(gm::GlobalMotion, dst, src) -> Matrix{Float64}

Residual local motion after removing the fitted global motion: for each correspondence, observed src - predicted src. Returns an n×2 matrix row-aligned with dst/src. Rows with large residuals indicate blocks moving inconsistently with the global model (independent motion, deformation, or bad codec vectors).

Note this is a sparse, block-based signal, not a dense optical-flow field.

fit_translation(dst, src) -> Matrix{Float64}

Robust (coordinate-wise median) translation fit. Returns a 2×3 transform.

fit_similarity(dst, src) -> Matrix{Float64}

Least-squares similarity (partial affine) fit: translation + rotation + uniform scale (Umeyama's method). Returns a 2×3 transform mapping dst points to src points.

fit_affine(dst, src) -> Matrix{Float64}

Least-squares full affine fit (translation, rotation, anisotropic scale, shear). Returns a 2×3 transform mapping dst points to src points.

ransac_fit(dst, src; model = :similarity, threshold = 2.0,
           max_iterations = 2000, confidence = 0.99,
           rng = Random.default_rng()) -> (A, inliers::BitVector)

Robustly fit a 2×3 transform mapping dst points to src points using RANSAC, then refine by refitting the model on all inliers.

• model: :translation, :similarity (translation + rotation + uniform scale), or :affine.
• threshold: inlier reprojection threshold in pixels.
• max_iterations: RANSAC iteration cap; iterations also stop early once the observed inlier ratio makes further sampling unnecessary at the requested confidence.

Returns the refined transform and the inlier mask (recomputed after refinement). Throws ArgumentError if there are fewer correspondences than the model's minimal sample size.

residuals(A, dst, src) -> Vector{Float64}

Euclidean residual per correspondence: ‖A*[dstᵢ,1] - srcᵢ‖.

apply_transform(A, p) -> (x, y)

Apply a 2×3 transform to the point p = (x, y).

transform_points(A, pts::AbstractMatrix) -> Matrix{Float64}

Apply a 2×3 transform to each row of an n×2 point matrix.

invert_transform(A) -> Matrix{Float64}

Invert a 2×3 affine transform.

compose_transform(B, A) -> Matrix{Float64}

Compose two 2×3 affine transforms: the result applies A first, then B, so apply_transform(compose_transform(B, A), p) == apply_transform(B, apply_transform(A, p)).

Useful for chaining per-frame global-motion estimates into a cumulative transform relative to an anchor frame (e.g. for stabilization): C_n = compose_transform(C_{n-1}, A_n).