Product Interaction: An Algebraic Formalism for Deep Learning Architectures

In this paper, we introduce product interactions, an algebraic formalism in which neural network layers are constructed from compositions of a multiplication operator defined over suitable algebras. Product interactions provide a principled way to generate and organize algebraic expressions by increasing interaction order. Our central observation is that algebraic expressions in modern neural networks admit a unified construction in terms of linear, quadratic, and higher-order product interactions. Convolutional and equivariant networks arise as symmetry-constrained linear product interactions, while attention and Mamba correspond to higher-order product interactions.

💡 Research Summary

The paper introduces a unifying algebraic framework called “product interaction” for constructing deep learning layers. The authors start by defining an algebra over a base field (ℝ or ℂ) as a vector space equipped with a bilinear product. Data features are embedded into a feature algebra A, while auxiliary structural information such as positions or time indices is embedded into a structural algebra B. By taking the tensor product B ⊗ A, a signal becomes a single algebraic element.

A key construct is the Multiplication operator (O_K), parameterized by a filter element (K) and two linear structural operators (L_1, L_2). It acts as (O_K(X)=L_1(K·L_2(X))), where “·” denotes the algebraic product. By composing several such operators, the authors define product interactions (O_{\text{prod}}). Two composition patterns are distinguished: (2) where the filter is the output of a previous multiplication, and (3) where the filter itself is fed as an input to another multiplication.

The notion of self‑interaction order quantifies how many times the input (X) appears multiplicatively in the expression. Order‑1 interactions correspond to linear operations such as standard convolutions, gated linear units, and classical state‑space models (SSMs). Order‑2 interactions arise when the filter is set equal to the input, yielding quadratic forms; this captures the input‑dependent gating of Mamba. Order‑3 interactions are obtained by nesting a quadratic interaction inside another multiplication, producing cubic forms that match the data‑dependent routing of attention mechanisms and SE(3)‑attention. Higher orders can be built by further nesting, enabling multi‑level message‑passing structures.

The symmetry principle (Definition 3.9) requires that a product interaction commute with any data transformation (T_g) representing a group action. This imposes constraints on the algebra’s structure constants, ensuring equivariance. The framework reproduces known equivariant architectures: translation‑equivariant convolutions (CNNs) as order‑1, SO(2)‑equivariant Harmonic Networks, SO(3)‑equivariant Tensor Field Networks, and SE(3)‑equivariant attention as higher‑order product interactions.

Empirical studies compare models of increasing self‑interaction order on image, sequence, and 3‑D point‑cloud tasks. Results show systematic performance gains when moving from order‑1 to order‑3, provided that appropriate symmetry constraints are retained. Higher order models also incur larger computational costs and risk over‑fitting if symmetry is not enforced. The paper demonstrates that structural operators (L_1, L_2) can implement masking, causal constraints, and neighborhood restrictions, further illustrating the flexibility of the formalism.

In summary, the authors present a single algebraic multiplication operator that, when composed, can generate a wide spectrum of modern deep learning architectures. By controlling self‑interaction order and symmetry constraints, the framework offers a principled way to navigate the design space of neural networks, bridging convolutional, attention‑based, state‑space, and equivariant models under one mathematical umbrella. This work opens avenues for systematic exploration of novel algebra‑driven architectures and for better understanding the trade‑offs between expressivity, symmetry, and efficiency.