EST: Towards Efficient Scaling Laws in Click-Through Rate Prediction via Unified Modeling

Efficiently scaling industrial Click-Through Rate (CTR) prediction has recently attracted significant research attention. Existing approaches typically employ early aggregation of user behaviors to maintain efficiency. However, such non-unified or partially unified modeling creates an information bottleneck by discarding fine-grained, token-level signals essential for unlocking scaling gains. In this work, we revisit the fundamental distinctions between CTR prediction and Large Language Models (LLMs), identifying two critical properties: the asymmetry in information density between behavioral and non-behavioral features, and the modality-specific priors of content-rich signals. Accordingly, we propose the Efficiently Scalable Transformer (EST), which achieves fully unified modeling by processing all raw inputs in a single sequence without lossy aggregation. EST integrates two modules: Lightweight Cross-Attention (LCA), which prunes redundant self-interactions to focus on high-impact cross-feature dependencies, and Content Sparse Attention (CSA), which utilizes content similarity to dynamically select high-signal behaviors. Extensive experiments show that EST exhibits a stable and efficient power-law scaling relationship, enabling predictable performance gains with model scale. Deployed on Taobao’s display advertising platform, EST significantly outperforms production baselines, delivering a 3.27% RPM (Revenue Per Mile) increase and a 1.22% CTR lift, establishing a practical pathway for scalable industrial CTR prediction models.

💡 Research Summary

The paper addresses the challenge of scaling click‑through‑rate (CTR) prediction models in industrial recommender systems, where models must process thousands of candidate items per request within strict latency budgets. Existing solutions either compress user behavior sequences early (hierarchical modeling) or only partially unify behavior and non‑behavioral features, leading to information loss or prohibitive computational cost due to dense self‑attention over long sequences.

The authors first analyze fundamental differences between CTR prediction and large language models (LLMs). They identify two key properties: (1) an asymmetry in information density—concise, high‑signal non‑behavioral features (the “query”) must interact with massive, low‑density behavior sequences (the “context”), making exhaustive behavior‑to‑behavior attention largely redundant; and (2) modality‑specific priors—user behaviors contain discrete IDs as well as rich content signals (images, text) that are more useful as similarity‑based relational cues than as raw token embeddings.

Guided by these insights, they propose the Efficiently Scalable Transformer (EST), a fully unified architecture that processes all raw inputs—non‑behavioral features N, user‑specific behaviors Bᵤ, candidate‑specific behaviors B𝚌, and associated multimodal content—in a single token sequence without any lossy aggregation. EST introduces two novel modules that run in parallel within each transformer layer:

1. Lightweight Cross‑Attention (LCA) – treats non‑behavioral tokens as queries and behavioral tokens as keys/values, thereby preserving the high‑impact cross‑feature interactions while pruning the low‑gain self‑interactions among behaviors. This dramatically reduces the quadratic cost associated with traditional self‑attention and decouples model scaling from sequence length.

2. Content Sparse Attention (CSA) – computes cosine similarity between content embeddings (e.g., image or text features) to identify a small subset of the most relevant behaviors for each query. Attention is then performed only on this subset, yielding a sparsity factor k ≪ L and an effective O(k·L) complexity. Because content similarity acts as a relational prior, CSA captures the most informative signals without incurring the overhead of full attention.

The overall EST layer consists of LCA, CSA, a feed‑forward network, and RMSNorm, stacked to any depth. Empirical studies on both public benchmarks and Alibaba’s internal Taobao logs demonstrate a stable power‑law scaling relationship: as model parameters increase from 10 M to 300 M, AUC, LogLoss, and business metrics improve consistently, outperforming hierarchical baselines (e.g., DIN, LONGER) and partially unified baselines (e.g., DeepFM, DIN‑Transformer).

A critical contribution is the real‑world deployment on Taobao’s display advertising platform. EST operates within a sub‑millisecond latency budget while handling behavior sequences of up to 1 k tokens. In online A/B tests, it delivers a 3.27 % increase in Revenue‑Per‑Mile (RPM) and a 1.22 % lift in CTR compared with the production baseline, confirming that the theoretical scaling gains translate into tangible revenue.

The authors acknowledge limitations: CSA’s effectiveness depends on the quality of content embeddings, and LCA may lose potency when non‑behavioral features are sparse. Future work is suggested on integrating pretrained multimodal encoders, learning adaptive behavior‑behavior interaction patterns, and exploring dynamic token sampling strategies.

In summary, EST leverages domain‑specific inductive biases—information‑density‑guided cross‑attention and content‑driven sparse attention—to achieve fully unified, computationally efficient CTR prediction that scales predictably with model size, and it provides the first industrial validation of scaling laws for recommender systems.