NDT: Non-Differential Transformer and Its Application to Sentiment Analysis

From customer feedback to social media, understanding human sentiment in text is central to how machines can interact meaningfully with people. However, despite notable progress, accurately capturing sentiment remains a challenging task, which continues to motivate further research in this area. To this end, we introduce Non-Differential Transformer (NDT). It is inspired by (but in contrast to) the state-of-the-art Differential Transformer (DT) model. While standard Transformers can struggle with irrelevant context, the sota DT model uses attention map subtraction, potentially for noise cancellation. We explore an alternative motivation, hypothesizing that benefits may arise from enabling different attention components to specialize on distinct concepts within the text, similar to multiplexing information channels or mixture models, rather than primarily canceling noise via subtraction. Guided by this concept-multiplexing (ConPlex) view, the specific architecture presented in this paper employs a purely additive strategy. It uses only positive weights, learned during training, to ensure constructive combination of these specialized attention perspectives. This design choice explores positive only integration, though our broader framework also shows promise with less constrained linear combinations involving both positive and negative weights. Our model computes attention via this positively weighted sum of multiple distinct attention maps. This allows the model to constructively integrate diverse signals and potentially capture more complex contextual relationships. Competitive performance is achieved by the proposed model for Sentiment Analysis while tested on multiple datasets. We conclude by presenting our results, challenges and future research agenda in this important area of research.

💡 Research Summary

The paper introduces the Non‑Differential Transformer (NDT), a novel architecture for sentiment analysis that departs from the subtractive attention mechanism of the Differential Transformer (DT). While DT computes attention as softmax(A₀) − λ·softmax(A₁) under a “noise‑cancellation” hypothesis, NDT is built on a new theoretical framework called Concept‑Multiplexing (ConPlex). ConPlex posits that effective sentiment understanding requires simultaneous processing of several conceptual channels—lexical sentiment cues, contextual modifiers, syntactic patterns, and domain‑specific signals. Accordingly, NDT aggregates multiple attention components additively:

Attention = Σᵢ λᵢ·softmax(QᵢKᵢᵀ/√d)·V,

where λ₀ is fixed to 1 and the remaining λᵢ are learnable scalars. Each component i has its own query (Qᵢ) and key (Kᵢ) projections (dimension d/2) but shares a common value projection V, which keeps parameter count modest while encouraging specialization. The λᵢ parameters are derived from two auxiliary vectors (λ_Qᵢ, λ_Kᵢ) via element‑wise multiplication, mean pooling, and a linear blend with component‑specific initialization (α_initᵢ) and bias (βᵢ). Four constraint regimes are explored for λᵢ: bounded positive