HHFT: Hierarchical Heterogeneous Feature Transformer for Recommendation Systems

We propose HHFT (Hierarchical Heterogeneous Feature Transformer), a Transformer-based architecture tailored for industrial CTR prediction. HHFT addresses the limitations of DNN through three key designs: (1) Semantic Feature Partitioning: Grouping heterogeneous features (e.g. user profile, item information, behaviour sequennce) into semantically coherent blocks to preserve domain-specific information; (2) Heterogeneous Transformer Encoder: Adopting block-specific QKV projections and FFNs to avoid semantic confusion between distinct feature types; (3) Hiformer Layer: Capturing high-order interactions across features. Our findings reveal that Transformers significantly outperform DNN baselines, achieving a +0.4% improvement in CTR AUC at scale. We have successfully deployed the model on Taobao’s production platform, observing a significant uplift in key business metrics, including a +0.6% increase in Gross Merchandise Value (GMV).

💡 Research Summary

The paper introduces HHFT (Hierarchical Heterogeneous Feature Transformer), a novel Transformer‑based architecture designed specifically for large‑scale click‑through‑rate (CTR) prediction in e‑commerce environments. Traditional deep neural network (DNN) models such as Wide & Deep, DeepFM, and DCNv2 dominate the field but suffer from two critical shortcomings: they cannot explicitly capture high‑order interactions among sparse, high‑dimensional features, and they treat heterogeneous inputs (user IDs, item attributes, query terms, behavior sequences) with shared parameters, leading to semantic confusion.

HHFT addresses these issues through three core innovations. First, Semantic Feature Partitioning groups raw inputs into semantically coherent blocks (e.g., user, item, query, behavior) and embeds each block with its own dimension‑specific embedding layer. Second, the Heterogeneous Transformer Encoder assigns each block its own query, key, value (QKV) projection matrices and feed‑forward network (FFN) parameters, preserving domain‑specific characteristics while still allowing cross‑block attention. Third, a Hiformer Layer sits atop the heterogeneous encoder; it concatenates all block tokens, applies a global composite projection, and then performs multi‑head attention on the transformed Q, K, V. This layer goes beyond pairwise attention, enabling the model to learn richer, higher‑order hierarchical interactions.

The overall pipeline consists of: (1) partitioning and embedding, (2) linear projection to a unified token dimension, (3) heterogeneous self‑attention, (4) Hiformer‑enhanced high‑order interaction, and (5) an MLP prediction head for CTR/CVR.

Experiments were conducted on Taobao’s massive real‑world dataset, encompassing billions of user‑item interactions. Offline evaluation used AUC as the primary metric, while online impact was measured by Gross Merchandise Value (GMV). HHFT (300 M parameters, 1.22 TFLOPs) outperformed a range of strong baselines—including DLRM‑MLP, DCNv2, AutoInt, the original Hiformer, Wukong, and RankMixer—by achieving a +0.008 absolute AUC gain over the DLRM‑MLP baseline. Notably, all Transformer‑based models surpassed DNN/FM‑based baselines, confirming the advantage of explicit attention for feature interaction modeling.

A detailed ablation study quantified the contribution of each component: replacing the DNN backbone with a vanilla Transformer yielded +0.0035 AUC; adding block‑specific QKV/FFN contributed +0.0018; incorporating the Hiformer layer added +0.0011; careful weight‑initialization and training tricks gave the largest single boost (+0.0040); and scaling the model size added +0.0034. Cumulatively, these improvements summed to a +0.0117 AUC gain relative to the plain MLP.

The authors also investigated scaling laws. By varying individual dense parameters while keeping others fixed, they observed that increasing width (token dimension of Transformer/Hiformer) consistently produced larger AUC improvements than increasing depth (number of layers). Moreover, scaling parameters associated with high‑order interactions (Hiformer token count and dimension) was more effective than scaling low‑order components (Transformer layer count). These findings provide practical guidance for resource‑constrained industrial deployments, suggesting that expanding token dimensions and high‑order modules yields the best return on investment.

A 30‑day online A/B test on Taobao’s search platform allocated 1 % of traffic to the HHFT model while the control group used the existing DNN ranker. HHFT achieved a +0.4 % absolute increase in CTR AUC and a +0.6 % lift in GMV. Given Taobao’s massive user base and transaction volume, these percentage gains translate into tens of billions of RMB in incremental revenue, demonstrating the model’s real‑world business impact.

In conclusion, HHFT successfully bridges the gap between heterogeneous feature handling and high‑order interaction modeling by combining semantic partitioning, block‑specific Transformer parameters, and a dedicated high‑order Hiformer layer. The work validates that Transformer‑based recommendation models obey predictable scaling laws, enabling systematic performance forecasting as model capacity grows. Future directions suggested include finer‑grained feature block definitions, multimodal extensions (e.g., image and text), and model compression techniques to further reduce latency while preserving the demonstrated accuracy gains.