Learning to relate images: Mapping units, complex cells and simultaneous eigenspaces

A fundamental operation in many vision tasks, including motion understanding, stereopsis, visual odometry, or invariant recognition, is establishing correspondences between images or between images and data from other modalities. We present an analysis of the role that multiplicative interactions play in learning such correspondences, and we show how learning and inferring relationships between images can be viewed as detecting rotations in the eigenspaces shared among a set of orthogonal matrices. We review a variety of recent multiplicative sparse coding methods in light of this observation. We also review how the squaring operation performed by energy models and by models of complex cells can be thought of as a way to implement multiplicative interactions. This suggests that the main utility of including complex cells in computational models of vision may be that they can encode relations not invariances.

💡 Research Summary

The paper tackles one of the most ubiquitous operations in computer vision – establishing correspondences between images – and argues that learning such relationships fundamentally requires multiplicative interactions, often called “mapping units.” The authors begin by noting that many vision tasks (optical flow, stereo, visual odometry, action recognition) depend not on the content of a single frame but on the transformation that links two frames. Traditional feature‑learning models (ICA, RBMs, auto‑encoders) are bipartite: hidden units receive a weighted sum of the input, which makes them well‑suited for representing static structure but ill‑suited for encoding a relation between two images.

A mapping unit introduces a three‑way product between an element of the first image (x_i), an element of the second image (y_j), and a latent “transform” variable (z_k). In matrix form the interaction can be written as y_j = Σ_i,k w_{ijk} x_i z_k, where the latent vector z encodes the transformation. This multiplicative structure allows the model to “modulate” the connections between the two images, effectively learning a mapping from one image to the other.

The central theoretical contribution is the observation that learning such mappings can be interpreted as detecting rotations in the simultaneous eigenspaces of a set of orthogonal matrices. If a family of orthogonal transformations shares a common set of eigenvectors, each transformation acts as a rotation (or reflection) within that subspace. The mapping units, through their multiplicative weights, learn to estimate the rotation angles associated with each eigen‑direction. Hence, the problem of learning image‑to‑image relationships reduces to learning a set of rotation parameters in a shared invariant subspace.

The paper then revisits classic energy models and the biological notion of complex cells. Energy models compute the sum of squared responses of two linear filters; mathematically this is equivalent to a product of the two filter responses, i.e., a special case of a multiplicative interaction. Complex cells, which are known for phase‑invariant responses, thus implement a form of multiplicative gating. The authors argue that the primary utility of complex cells in computational vision may not be invariance per se, but rather the ability to encode relationships (e.g., motion direction, disparity) through the squaring non‑linearity.

A broad survey of recent multiplicative sparse‑coding approaches follows. The authors discuss Gated Boltzmann Machines, bilinear models, Independent Subspace Analysis (ISA), Adaptive Subspace SOM (ASSOM), and various “routing‑circuit” or “dynamic mapping” architectures. All these models share the core idea of a latent multiplicative code that disentangles two sources of variability (often termed “style” and “content”) or directly captures the multitude of possible transformations between image pairs. The paper highlights how many of these methods can be re‑interpreted as learning rotations in shared eigenspaces, thereby unifying a disparate literature under a common mathematical lens.

Although the paper does not present new experimental results, it references extensive prior work showing that multiplicative models excel at optical flow estimation, stereo matching, and action recognition, especially when the squaring non‑linearity (or cross‑product) is employed. The authors predict that any vision task that requires representing the original information of a video beyond static appearance will benefit from such multiplicative mechanisms.

In conclusion, the work positions multiplicative interactions as the essential computational primitive for relational vision. By linking mapping units to rotations in simultaneous eigenspaces and by re‑interpreting complex‑cell energy models as approximate multiplicative gates, the paper provides a fresh theoretical justification for why biologically inspired squaring non‑linearities improve relational tasks. This insight opens avenues for designing deeper architectures that learn richer transformation manifolds, extending beyond simple translations to arbitrary linear (and potentially non‑linear) transformations, and even to multimodal relationships such as image‑text or image‑audio correspondences.