Magnitude Distance: A Geometric Measure of Dataset Similarity

Quantifying the distance between datasets is a fundamental question in mathematics and machine learning. We propose \textit{magnitude distance}, a novel distance metric defined on finite datasets using the notion of the \emph{magnitude} of a metric space. The proposed distance incorporates a tunable scaling parameter, $t$, that controls the sensitivity to global structure (small $t$) and finer details (large $t$). We prove several theoretical properties of magnitude distance, including its limiting behavior across scales and conditions under which it satisfies key metric properties. In contrast to classical distances, we show that magnitude distance remains discriminative in high-dimensional settings when the scale is appropriately tuned. We further demonstrate how magnitude distance can be used as a training objective for push-forward generative models. Our experimental results support our theoretical analysis and demonstrate that magnitude distance provides meaningful signals, comparable to established distance-based generative approaches.

💡 Research Summary

The paper introduces magnitude distance, a novel dissimilarity measure for finite datasets derived from the concept of magnitude of a metric space, originally defined by Leinster. The authors first extend the definition of magnitude to collections that may contain duplicate points, proving that the magnitude of a multiset equals the magnitude of its set of distinct elements (Theorem 4.1). Using this, they define the magnitude distance between two finite sets X and Y as

  dₜᴹᵃᵍ(X,Y) = 2·Mag(X∪Y; t) − Mag(X; t) − Mag(Y; t),

where t > 0 is a scaling parameter and Mag(·; t) is computed from the similarity matrix ζₓᵧ = exp(−t·d(x,y)). The scaling parameter controls the emphasis of the distance: small t aggregates global structure, while large t highlights local differences and sample variability.

Theoretical properties are explored in depth. The distance is symmetric and non‑negative by construction. An identity‑of‑indiscernibles property holds under a newly introduced notion of magnitude equivalence: dₜᴹᵃᵍ(X,Y)=0 iff X and Y have the same support of non‑zero magnitude weights at scale t. However, the triangle inequality fails in dimensions D > 1, so dₜᴹᵃᵍ is not a metric in the strict sense. The authors analyze the behavior of dₜᴹᵃᵍ as t varies: it converges to 0 as t → 0 and to the cardinality of the symmetric difference |X Δ Y| as t → ∞ (Theorem 5.3). Moreover, for any intermediate value α∈(0,|X Δ Y|) there exists a t achieving dₜᴹᵃᵍ=α, showing that the distance can be tuned to any desired sensitivity regardless of ambient dimension (Proposition 5.4).

A key contribution is the comparison with the Maximum Mean Discrepancy (MMD). Both use the same exponential kernel kₜ(x,y)=exp(−t·d(x,y)), but MMD aggregates the kernel matrix via a quadratic form (1ᵀK1), which in high dimensions suffers from spectral degeneracy: the kernel matrix becomes low‑rank, and the statistic collapses toward zero. In contrast, magnitude distance depends on the inverse kernel matrix (sum of all entries of K⁻¹), thereby incorporating the full spectrum and remaining stable even when most eigenvalues are near zero. Empirical results (Figure 2) demonstrate that while MMD quickly vanishes as dimension grows, magnitude distance with appropriately scaled t (e.g., t ≈ 1/√D) stays roughly constant.

The authors then apply magnitude distance as a training objective for a push‑forward generative model, termed Magnitude Generative Network (MagGN). Inspired by curriculum learning, MagGN starts with a small t, allowing the generator to capture coarse, global structure, and gradually increases t so that finer details are learned later. Experiments on synthetic Gaussian mixtures and higher‑dimensional image data show that MagGN achieves comparable or better sample quality than MMD‑GAN and Wasserstein‑GAN, especially in regimes where traditional distances lose discriminative power.

Strengths of the work include: (1) a principled, mathematically grounded distance that can be tuned across scales; (2) theoretical guarantees about its limiting behavior and robustness to high dimensionality; (3) an elegant connection to kernel methods that clarifies why it outperforms MMD in certain settings; (4) a practical demonstration that the distance can serve as a loss for generative modeling.

Limitations are also acknowledged. The lack of the triangle inequality means dₜᴹᵃᵍ cannot be directly used in algorithms that rely on metric properties (e.g., k‑nearest‑neighbors, metric‑based clustering). Computing the inverse of the similarity matrix incurs O(n³) time and O(n²) memory, which is prohibitive for large datasets; the paper does not provide scalable approximations, leaving this as an open problem. Moreover, the analysis is confined to Euclidean spaces; extending the framework to non‑Euclidean domains such as graphs or manifolds would require additional work.

Future directions suggested by the authors include developing randomized or low‑rank approximations (Nyström, sketching) to make magnitude distance tractable for big data, exploring alternative kernels beyond the exponential, and adapting the notion of magnitude to structured spaces (e.g., point clouds on manifolds, network data). Investigating theoretical connections with other scale‑space concepts (diffusion distances, heat kernels) could also deepen understanding.

In summary, the paper presents a compelling new tool—magnitude distance—that bridges abstract categorical notions of size with concrete statistical tasks. By offering a tunable, high‑dimensional‑robust measure and demonstrating its utility in generative modeling, it opens a promising line of research at the intersection of metric geometry, kernel methods, and deep learning.