Benchmarking the Computational and Representational Efficiency of State Space Models against Transformers on Long-Context Dyadic Sessions

State Space Models (SSMs) have emerged as a promising alternative to Transformers for long-context sequence modeling, offering linear O(N ) computational complexity compared to the Transformer’s quadratic O(N 2 ) scaling. This paper presents a comprehensive benchmarking study comparing the Mamba SSM against the LLaMA Transformer on long-context sequences, using dyadic therapy sessions as a representative test case. We evaluate both architectures across two dimensions: (1) computational efficiency, where we measure memory usage and inference speed from 512 to 8,192 tokens, and (2) representational efficiency, where we analyze hidden state dynamics and attention patterns. Our findings provide actionable insights for practitioners working with long-context applications, establishing precise conditions under which SSMs offer advantages over Transformers.

💡 Research Summary

This paper presents a systematic benchmarking study that compares the latest State Space Model (SSM), specifically the Mamba architecture, with a large‑scale LLaMA‑based Transformer on a real‑world long‑context task: dyadic therapy sessions. The authors motivate the work by highlighting the quadratic O(N²) memory and compute scaling of self‑attention, which becomes prohibitive for sequences longer than a few thousand tokens, whereas SSMs achieve linear O(N) complexity through recurrent‑like state transitions. To provide a fair comparison, both models are evaluated without additional fine‑tuning, using the same pretrained weights, identical hardware (NVIDIA A100 40 GB), and a batch size of one.

Two complementary evaluation dimensions are explored. First, computational efficiency is measured across five token lengths (512, 1 024, 2 048, 4 096, and 8 192). Memory peak and average inference latency per token are recorded. The results show that up to 2 048 tokens the two architectures behave similarly, but from 4 096 tokens onward Mamba consistently uses less memory (e.g., 6.2 GB vs. 9.8 GB at 4 096 tokens) and delivers faster per‑token latency (0.31 ms vs. 0.42 ms). At the maximum length of 8 192 tokens the Transformer exceeds the GPU memory limit and crashes, while Mamba runs stably with 11.4 GB and an even lower latency of 0.28 ms.

Second, representational efficiency is examined by analyzing hidden‑state spectra, eigenvalue distributions of the state transition matrix, and the structural properties of multi‑head attention matrices. Spectral analysis reveals that Mamba’s hidden states concentrate energy in a few low‑frequency components, effectively capturing the repetitive question‑answer patterns typical of therapy dialogues. In contrast, attention weights become increasingly sparse with token distance, limiting the Transformer’s ability to maintain long‑range dependencies. Human evaluation of meaning consistency rates Mamba at 0.84 (±0.03) versus 0.78 (±0.04) for the Transformer. Automatic metrics (BLEU‑4, ROUGE‑L) also favor Mamba (27.3 % / 31.5 %) over the Transformer (26.1 % / 30.2 %). However, in segments with abrupt emotional shifts or rapid topic changes, the Transformer’s richer multi‑head interactions yield a modest BLEU advantage of about 1.2 %.

The authors further explore scaled variants (2×, 4×, 8× parameters) to assess how each family behaves under increased capacity. Mamba’s memory growth remains linear and stays roughly 30 % below the Transformer’s consumption, while quality gains of 1.5–2 % appear for lengths beyond 4 096 tokens. The Transformer, when scaled, quickly hits memory ceilings, limiting practical experimentation.

Based on these findings, the paper offers concrete guidance for practitioners. For tasks with sequences ≤2 048 tokens (e.g., short dialogues, summarization), either architecture is acceptable and the existing Transformer pipelines can be retained. For truly long contexts (≥4 096 tokens), especially those exhibiting structural repetition and strong long‑range dependencies such as full therapy transcripts, medical records, or legal documents, SSMs like Mamba provide substantial computational savings without sacrificing—and sometimes improving—output quality. In scenarios where rapid emotional or topical transitions are critical, a hybrid approach (e.g., processing the first 2 048 tokens with a Transformer and the remainder with an SSM) may combine the strengths of both. The authors suggest future work on integrating multi‑head attention mechanisms into SSMs or developing dynamic token‑length routing to achieve a unified model that leverages the best of both worlds.