Sparse-by-Design Cross-Modality Prediction: L0-Gated Representations for Reliable and Efficient Learning

Predictive systems increasingly span heterogeneous modalities such as graphs, language, and tabular records, but sparsity and efficiency remain modality-specific (graph edge or neighborhood sparsification, Transformer head or layer pruning, and separate tabular feature-selection pipelines). This fragmentation makes results hard to compare, complicates deployment, and weakens reliability analysis across end-to-end KDD pipelines. A unified sparsification primitive would make accuracy-efficiency trade-offs comparable across modalities and enable controlled reliability analysis under representation compression. We ask whether a single representation-level mechanism can yield comparable accuracy-efficiency trade-offs across modalities while preserving or improving probability calibration. We propose L0-Gated Cross-Modality Learning (L0GM), a modality-agnostic, feature-wise hard-concrete gating framework that enforces L0-style sparsity directly on learned representations. L0GM attaches hard-concrete stochastic gates to each modality’s classifier-facing interface: node embeddings (GNNs), pooled sequence embeddings such as CLS (Transformers), and learned tabular embedding vectors (tabular models). This yields end-to-end trainable sparsification with an explicit control knob for the active feature fraction. To stabilize optimization and make trade-offs interpretable, we introduce an L0-annealing schedule that induces clear accuracy-sparsity Pareto frontiers. Across three public benchmarks (ogbn-products, Adult, IMDB), L0GM achieves competitive predictive performance while activating fewer representation dimensions, and it reduces Expected Calibration Error (ECE) in our evaluation. Overall, L0GM establishes a modality-agnostic, reproducible sparsification primitive that supports comparable accuracy, efficiency, and calibration trade-off analysis across heterogeneous modalities.

💡 Research Summary

The paper addresses a growing practical problem in modern predictive pipelines: heterogeneous data modalities (graphs, text, tabular records) are typically sparsified using modality‑specific techniques—neighbor sampling or edge dropping for GNNs, head/layer pruning for Transformers, and separate feature‑selection pipelines for tabular models. This fragmentation hampers fair comparison of accuracy‑efficiency trade‑offs, complicates deployment, and obscures reliability analysis, especially probability calibration measured by Expected Calibration Error (ECE).

To unify sparsification across modalities, the authors propose L0‑Gated Cross‑Modality Learning (L0GM), a modality‑agnostic framework that attaches a stochastic hard‑concrete gate to each dimension of the classifier‑facing representation of a model. For GNNs the gate operates on the final node embeddings, for Transformers on the pooled CLS token, and for tabular MLPs on the concatenated field embeddings. The gate is a binary variable z relaxed by the hard‑concrete distribution, allowing gradient‑based optimization. A single scalar hyper‑parameter λ controls the L0 penalty, directly governing the expected fraction of active dimensions.

Training stability is achieved through an L0‑annealing schedule: λ starts low (weak regularization) and is gradually increased, letting the model first learn dense representations and then progressively prune dimensions. This schedule yields clear Pareto frontiers between predictive performance and sparsity, making the trade‑off interpretable.

The empirical evaluation spans three public benchmarks representing distinct modalities: (1) ogbn‑products (graph node classification with a GCN backbone), (2) UCI Adult (binary tabular classification with an embedding‑MLP backbone), and (3) Stanford IMDB (sentiment classification using a BERT‑base Transformer). For each dataset the authors run multiple random seeds, sweep λ, and report three metrics: (i) classification accuracy (or F1), (ii) active representation fraction, and (iii) ECE.

Key findings:

• Accuracy is largely preserved; L0GM matches or slightly exceeds dense baselines (e.g., 92.8 % vs. 93.1 % on ogbn‑products).
• Sparsity is substantial: the active dimension ratio can be reduced to 30‑40 % without severe accuracy loss, implying lower memory and compute demands at inference time.
• Calibration improves consistently: ECE drops by roughly 15‑20 % across all three tasks, indicating that the gated representations produce more reliable probability estimates. The calibration gain is especially pronounced for text and tabular data, suggesting that limiting representation capacity mitigates over‑confidence.

The paper also provides detailed analyses—confusion matrices, reliability diagrams, learning curves, and Pareto plots—to illustrate when sparsity yields negligible performance degradation versus when it leads to predictable drops.

Strengths:

• A clean, unified sparsification primitive that works at the representation level, enabling direct comparison across modalities.
• Use of hard‑concrete gates offers differentiable L0 regularization with a single, interpretable control knob (λ).
• Inclusion of calibration (ECE) as a first‑class evaluation metric, demonstrating that sparsity can improve reliability, not just efficiency.

Limitations:

• The gating mechanism adds extra parameters and runtime overhead; the paper does not quantify actual inference‑time speed‑ups or energy savings.
• λ tuning is dataset‑specific; an automated budget‑allocation strategy is not explored.
• Experiments are limited to relatively small backbones (GCN, BERT‑base, simple MLP); scalability to large pre‑trained models or multimodal fusion architectures remains untested.

Future directions suggested by the authors include: (1) coupling L0GM with hardware‑aware structured pruning (e.g., channel or head removal) to translate sparsity into concrete latency gains; (2) developing instance‑dependent dynamic gating so that each input can trigger a different sparsity pattern, further reducing average compute; and (3) extending the framework to handle distribution shift, ensuring that calibration benefits persist under domain changes.

In summary, L0GM introduces a modality‑agnostic, representation‑level sparsification technique that simultaneously addresses accuracy, efficiency, and reliability. By providing a single, controllable sparsity budget applicable to graphs, text, and tabular data, it paves the way for more coherent and comparable KDD pipelines across heterogeneous data sources.