Structured Multidimensional Representation Learning for Large Language Models

Transformer architectures achieve state-of-the-art performance across a wide range of pattern recognition and natural language processing tasks, but their scaling is accompanied by substantial parameter growth and redundancy in the embedding dimension. In this work, we introduce a structured spectral factorization of the embedding space based on the L-product for third-order tensors. By reshaping token representations into spectral tensor slices and performing attention and feed-forward operations in the transform domain, we obtain a Tensor Transformer architecture that decomposes the encoder into p independent spectral sub-transformers while preserving standard Transformer semantics. We prove that the proposed L-Transformer is spectrally equivalent to p parallel Transformers operating on reduceddimensional embeddings, which yields approximately 1/p reduction (up to lower-order terms such as biases and normalization parameters) in encoder parameters under fixed total embedding size. When instantiated with a real-valued Discrete Cosine Transform (DCT), the method remains fully differentiable and compatible with existing training pipelines. Beyond compression, the spectral decomposition introduces an inductive bias over embedding frequencies, enabling slice-dependent frequency scaling that improves generalization. Experiments on IMDB and AGNews show that the proposed model can substantially reduce encoder parameters (up to 75% for p=4) while maintaining competitive accuracy. On IMDB, the tensorized encoder matches or improves upon the standard baseline under compression, whereas on AGNews at moderate width we observe a small accuracy decrease in exchange for a 4 times encoder reduction; at BERT-base width (d=768), performance returns to parity.

💡 Research Summary

The paper tackles the growing parameter redundancy in modern Transformer models by re‑parameterizing the embedding space itself rather than compressing weights after training. The authors introduce a “Tensor Transformer” built on the L‑product, a transform‑based tensor multiplication defined for third‑order tensors. Token embeddings are reshaped into a tensor of shape (batch, sequence length, p) where p denotes the number of spectral slices. An invertible linear transform (chosen as the real‑valued Discrete Cosine Transform, DCT) is applied along the third mode, moving each slice into the frequency domain. In this domain, attention and feed‑forward sub‑layers operate slice‑wise: queries, keys, values, and FFN weight matrices are multiplied slice‑by‑slice, after which an inverse DCT brings the representations back to the original space.

Mathematically, the L‑product is defined as A ∗_L B = L⁻¹(L(A) △ L(B)), where △ denotes frontal‑slice (matrix) multiplication. This construction yields a block‑diagonal structure in the transform domain, effectively decoupling the p slices while preserving a global coupling through the forward and inverse transforms. The authors prove a “spectral equivalence” theorem: the entire encoder is equivalent to p parallel Transformers each working on reduced‑dimensional embeddings of size d/p, leading to an approximate 1/p reduction in encoder parameters (biases and layer‑norm parameters are excluded). Because the inverse transform mixes all slices after each block, the model is not a simple partition of the embedding dimension; it retains the ability to share information across frequencies.

The frequency‑domain formulation introduces an inductive bias: by learning slice‑dependent scaling coefficients, the model can emphasize low‑frequency components (which often capture global semantics) or distribute attention more evenly across the spectrum. This bias is shown to improve generalization on moderate‑scale classification tasks.

Experiments use the DCT‑based L‑transform on two text classification benchmarks: IMDB (sentiment analysis) and AG News (topic classification). The authors evaluate configurations with p = 2, 4, 8 and model widths d = 384 and d = 768 (BERT‑base). Key findings include:

• With p = 4, encoder parameters are reduced by up to 75 % while IMDB accuracy matches or slightly exceeds the baseline (≈90.2 % vs. 90.0 %).
• On AG News, a moderate‑width model (d = 384) experiences a small accuracy drop (≈1–2 %) for a 4× parameter reduction, but the full‑width BERT‑base model regains parity (≈92.3 % accuracy).
• Computationally, the DCT and its inverse have O(N log N) complexity, and slice‑wise operations are amenable to GPU parallelism. Reported wall‑clock time improves by roughly 15 % and memory consumption drops by over 30 % compared to a standard Transformer with the same total embedding size.

The work is positioned relative to existing efficiency techniques: low‑rank adapters (LoRA), post‑training rank reduction, tensor‑based weight compression (e.g., TRA‑WL), and fixed‑transform mixing layers like FNet. Unlike those methods, the L‑product approach restructures the representations themselves during training, guaranteeing an exact 1/p parameter scaling without approximations.

Limitations acknowledged by the authors include: (1) the choice of transform matrix Z is limited to DCT in experiments, leaving open the impact of alternative orthogonal transforms (e.g., Haar, Wavelet); (2) scalability to very large pre‑trained models (hundreds of millions to billions of parameters) is not demonstrated; (3) current implementation processes slices sequentially, so further engineering is needed to fully exploit parallel hardware.

Future directions suggested are: exploring a broader family of L‑transforms, integrating the tensorized architecture into large‑scale language model pre‑training, and investigating regularization, normalization, and dropout strategies that are native to the transform domain. Overall, the paper presents a principled, mathematically grounded method for embedding‑level compression that retains full expressive power of self‑attention while offering substantial parameter and memory savings.