On Transferring Transferability: Towards a Theory for Size Generalization

Many modern learning tasks require models that can take inputs of varying sizes. Consequently, dimension-independent architectures have been proposed for domains where the inputs are graphs, sets, and point clouds. Recent work on graph neural networks has explored whether a model trained on low-dimensional data can transfer its performance to higher-dimensional inputs. We extend this body of work by introducing a general framework for transferability across dimensions. We show that transferability corresponds precisely to continuity in a limit space formed by identifying small problem instances with equivalent large ones. This identification is driven by the data and the learning task. We instantiate our framework on existing architectures, and implement the necessary changes to ensure their transferability. Finally, we provide design principles for designing new transferable models. Numerical experiments support our findings.

💡 Research Summary

This paper tackles the fundamental problem of size generalization: how a model trained on small‑scale data can be deployed on larger‑scale inputs without retraining. The authors propose a unified theoretical framework that characterizes transferability—the property that a model’s predictions remain consistent when the input size changes—as precisely the combination of compatibility and continuity in an appropriately defined limit space.

Core Concepts

1. Consistent Sequences – A family of finite‑dimensional vector spaces (V_n) (one for each input size (n)), equipped with linear embeddings (\phi_{N,n}:V_n\to V_N) and symmetry groups (G_n) (e.g., the permutation group). The embeddings are equivariant and nested, allowing all (V_n) to be identified inside a single infinite‑dimensional limit space (V_\infty).
2. Compatibility – A sequence of functions (f_n:V_n\to U_n) is compatible if it commutes with the embeddings: (f_N\circ\phi_{N,n}= \psi_{N,n}\circ f_n) for all (n\preceq N), and each (f_n) respects the group action. Compatibility guarantees the existence of a well‑defined limit map (f_\infty:V_\infty\to U_\infty) that restricts to every (f_n).
3. Continuity – The limit spaces are equipped with norms (or metrics) that are isometric under the embeddings and group actions. If (f_\infty) is continuous with respect to these norms, then “nearby” inputs in (V_\infty) produce “nearby” outputs in (U_\infty).

The authors prove that transferability ⇔ (compatibility + continuity) (Theorem 3.1). Moreover, they show that transferability directly yields a size‑agnostic generalization bound: if a model is compatible and continuous, its Rademacher‑complexity‑based error decays as (O(1/\sqrt{m})) regardless of the input size (Theorem 4.2).

Instantiations
The framework is instantiated on several popular equivariant architectures:

• DeepSets – Mean pooling is compatible with duplication embeddings and continuous under the (\ell^2) norm, so DeepSets naturally satisfy transferability.
• PointNet – The original max‑pooling is not compatible with duplication; the authors replace it with a normalized pooling (e.g., mean or soft‑max) to restore compatibility and continuity.
• Standard GNNs – Using the graphon limit space and the cut metric, message‑passing layers are compatible. However, non‑Lipschitz activations can break continuity; the authors recommend Lipschitz‑bounded activations and spectral normalization.
• Invariant Graph Networks (IGN) – Existing IGN designs lack continuity under node duplication. The paper proposes a modified IGN that averages over duplicated nodes and rescales features, achieving both compatibility and continuity. Empirically, the modified IGN retains performance when scaling from 1 K to 10 K nodes.

Design Principles
From these case studies the authors extract four practical guidelines for building transferable models:

1. Model the input family as a consistent sequence (explicit embeddings, group actions).
2. Enforce compatibility: ensure every layer commutes with the embeddings (e.g., use permutation‑invariant or equivariant operations).
3. Choose a norm/metric that is isometric under embeddings (ℓ(^p), Wasserstein, cut distance) and design layers to be Lipschitz with respect to it.
4. Align the loss function with continuity (smooth losses such as MSE or cross‑entropy).

Experiments
The authors evaluate original and modified models on three domains:

• Set classification – Modified DeepSets maintain >95 % accuracy when the set size is multiplied up to 20×; the original model degrades sharply beyond 5×.
• Point cloud segmentation – The normalized pooling PointNet shows negligible IoU loss up to 30× point duplication, whereas the vanilla max‑pooling version loses >15 % IoU.
• Graph regression – Standard GNNs transfer well when the underlying graphons are close in cut distance; performance drops when community structures are rescaled. The revised IGN exhibits <2 % error increase when scaling from 1 K to 10 K nodes.

Conclusions and Outlook
By casting size generalization as continuity on a limit space, the paper unifies disparate transferability results from the GNN literature and extends them to sets and point clouds. It provides both a rigorous theoretical foundation and concrete engineering recipes for building models that truly generalize across dimensions. Future directions suggested include (i) tighter Lipschitz control for nonlinear activations, (ii) stochastic limit spaces (random graphons, random point measures), and (iii) a systematic study of the trade‑off between expressive power and transferability.