An Improved Observation Model for Super-Resolution under Affine Motion

Super-resolution (SR) techniques make use of subpixel shifts between frames in an image sequence to yield higher-resolution images. We propose an original observation model devoted to the case of non isometric inter-frame motion as required, for instance, in the context of airborne imaging sensors. First, we describe how the main observation models used in the SR literature deal with motion, and we explain why they are not suited for non isometric motion. Then, we propose an extension of the observation model by Elad and Feuer adapted to affine motion. This model is based on a decomposition of affine transforms into successive shear transforms, each one efficiently implemented by row-by-row or column-by-column 1-D affine transforms. We demonstrate on synthetic and real sequences that our observation model incorporated in a SR reconstruction technique leads to better results in the case of variable scale motions and it provides equivalent results in the case of isometric motions.

💡 Research Summary

Super‑resolution (SR) aims to reconstruct a high‑resolution (HR) image from a sequence of low‑resolution (LR) frames by exploiting sub‑pixel displacements between the frames. Most SR literature assumes that inter‑frame motion is isometric – i.e., composed of pure translations, rotations, or uniform scaling – and builds observation models accordingly. However, many practical imaging scenarios, such as airborne or UAV platforms, involve non‑isometric affine motion where the scale and shear components change from frame to frame. Under such conditions, conventional observation models either approximate the affine transform poorly or introduce significant interpolation errors, leading to blurred edges, loss of fine texture, and overall degradation of the reconstructed HR image.

The authors first review the dominant observation models, focusing on the Elad‑Feuer model, which treats motion by applying a 2‑D interpolation kernel after warping each LR frame onto the HR grid. While mathematically sound for small, near‑isometric motions, this approach suffers from two major drawbacks when affine motion is present: (1) the warped sampling points rarely align with the discrete HR lattice, causing aliasing; (2) the computational cost grows quadratically with the image size because a full 2‑D interpolation must be performed for every pixel.

To overcome these limitations, the paper proposes a novel observation model specifically designed for affine motion. The key insight is that any 2‑D affine transformation can be decomposed into a sequence of two shear (or “skew”) operations and a uniform scaling. Shear transformations act along a single axis – either rows or columns – and therefore can be implemented as a series of 1‑D affine transforms. By processing the image row‑by‑row for the first shear, then column‑by‑column for the second shear, and finally applying a global scaling, the model retains the exact geometric effect of the original affine matrix while allowing the use of highly optimized 1‑D interpolation kernels (e.g., Lanczos, cubic spline).

This decomposition yields several practical advantages:

1. Computational Efficiency – Each shear step requires only O(N) operations (N = number of pixels), dramatically reducing runtime compared with full 2‑D interpolation.
2. Memory Locality – Row‑wise and column‑wise passes lead to contiguous memory accesses, making the algorithm well‑suited for GPU acceleration and cache‑friendly CPU implementations.
3. Interpolation Accuracy – Because the sampling points after each 1‑D shear lie on a regular 1‑D grid, high‑quality 1‑D kernels can be applied without the need for costly 2‑D resampling, minimizing aliasing and preserving high‑frequency details.
4. Flexibility – The model naturally accommodates frame‑dependent affine parameters; each frame’s shear matrices are computed independently, enabling real‑time processing of dynamic scenes where scale and shear vary rapidly.

The authors embed the new observation model into a standard SR reconstruction framework that solves a regularized inverse problem (typically via iterative back‑projection or a Bayesian MAP estimator). They evaluate the method on both synthetic data, where ground‑truth HR images are known, and real airborne video sequences. In synthetic experiments with scale variations of ±10 % and ±20 %, the proposed model improves peak signal‑to‑noise ratio (PSNR) by an average of 1.2 dB and structural similarity index (SSIM) by 0.03 relative to the baseline Elad‑Feuer model. Visual inspection confirms sharper edges and better texture fidelity. In real data, the method produces clearer building outlines and more detailed ground textures, especially in regions where the camera’s zoom or altitude changes between frames. When the motion is purely isometric, the new model yields results statistically indistinguishable from the traditional approach, confirming that it does not sacrifice performance in the simpler case.

In summary, the paper delivers a mathematically rigorous yet computationally lightweight observation model that accurately captures affine motion in SR problems. By leveraging the shear‑decomposition of affine matrices, it sidesteps the pitfalls of 2‑D interpolation, reduces computational load, and enhances reconstruction quality for applications with variable scale and shear – notably remote sensing, UAV surveillance, and medical imaging where zoom or perspective changes are common. The authors suggest future extensions to handle full perspective (projective) transforms, multi‑camera arrays, and integration with deep‑learning‑based SR networks to combine the interpretability of model‑based methods with the representational power of neural networks.