Exploring the Upper Limits of Text-Based Collaborative Filtering Using Large Language Models: Discoveries and Insights

Text-based collaborative filtering (TCF) has emerged as the prominent technique for text and news recommendation, employing language models (LMs) as text encoders to represent items. However, the current landscape of TCF models mainly relies on the utilization of relatively small or medium-sized LMs. The potential impact of using larger, more powerful language models (such as these with over 100 billion parameters) as item encoders on recommendation performance remains uncertain. Can we anticipate unprecedented results and discover new insights? To address this question, we undertake a comprehensive series of experiments aimed at exploring the performance limits of the TCF paradigm. Specifically, we progressively augment the scale of item encoders, ranging fromone hundred million to one hundred billion parameters, in order to reveal the scaling limits of the TCF paradigm. Moreover, we investigate whether these exceptionally large LMs have the potential to establish a universal item representation for the recommendation task, thereby revolutionizing the traditional ID paradigm, which is considered a significant obstacle to developing transferable “one model fits all” recommender models. Our study not only demonstrates positive results but also uncovers unexpected negative outcomes, illuminating the current state of the TCF paradigm within the community. These findings will evoke deep reflection and inspire further research on text-based recommender systems.

💡 Research Summary

This paper, “Exploring the Upper Limits of Text-Based Collaborative Filtering Using Large Language Models: Discoveries and Insights,” presents a comprehensive empirical investigation into the potential and limitations of scaling up language models (LMs) as item encoders within the Text-based Collaborative Filtering (TCF) paradigm. The core motivation stems from the observation that current TCF models primarily rely on small or medium-sized LMs (e.g., BERT), leaving the impact of using extremely large LMs (LLMs) with hundreds of billions of parameters largely unknown. The study aims to answer four pivotal questions: (Q1) the performance scaling limits of TCF with increasingly larger item encoders; (Q2) whether such LLMs can generate universal item representations for recommendation; (Q3) if TCF with a massive LLM encoder can surpass the simple yet powerful ID-based Collaborative Filtering (IDCF) paradigm, especially for non-cold-start (warm) item recommendation; and (Q4) how close TCF is to achieving a transferable “one model fits all” universal recommendation system.

The authors conduct extensive experiments on three real-world text recommendation datasets: MIND (news), HM (fashion), and Bili (video comments). They employ two foundational recommendation backbones—the two-tower DSSM model and the sequential SASRec model—and progressively scale the item encoder from 160 million parameters (BERT-large) up to 175 billion parameters (GPT-3), utilizing models like LLaMA at intermediate scales (7B, 13B, 30B, 65B). The evaluation focuses on top-N recommendation metrics like Recall and NDCG.

The key findings are multifaceted and offer crucial insights for the field:

1. Diminishing Returns on Scale (Q1): While increasing the size of the item encoder consistently improves recommendation performance, the gains exhibit clear diminishing returns. Significant improvements are observed when scaling from ~100M to ~10B parameters, but the performance curve flattens considerably beyond the ~100B parameter scale. This suggests a performance ceiling for TCF where scaling the item encoder alone is insufficient, likely because the user modeling component becomes a bottleneck in capturing complex user-item interaction patterns.

2. The Persistent Superiority of IDCF in Warm Scenarios (Q3): A striking and critical finding is that even the TCF model equipped with a 175B-parameter LLM as its item encoder could not consistently outperform the simple IDCF model in warm-item recommendation settings. IDCF, leveraging minimal item ID embeddings, proves exceptionally efficient at encoding the intricate collaborative signals latent in user interaction histories. This underscores a fundamental challenge for pure content-based approaches and explains why ID features remain entrenched in industrial systems.

3. Potential for Universal Representation and Transfer (Q2, Q4): On a positive note, the textual embeddings extracted from large pre-trained LLMs (particularly the 175B model) showed consistent quality across diverse domains, indicating their potential as rich, general-purpose semantic item representations. Furthermore, a TCF model pre-trained on a large-scale dataset (Bili8M) demonstrated non-trivial zero-shot recommendation capability on other datasets and could be effectively fine-tuned with small amounts of target data. This validates the transfer learning potential of the TCF paradigm, a step towards more generalizable systems.

4. Practical Implications and Future Directions: The study provides a sobering yet constructive reality check. It concludes that simply scaling the item encoder to extreme sizes is not a panacea for surpassing the ID-based paradigm in its core strength. The path forward likely involves hybrid approaches that intelligently combine ID-based collaborative signals with LLM-powered semantic features. Moreover, future research should focus on novel architectures that better bridge the gap between high-quality text understanding and effective modeling of user interaction patterns, rather than solely pursuing scale.

In summary, this paper delivers valuable empirical evidence and nuanced insights by pushing the TCF paradigm to its scaling limits. It highlights both the promising transferability and semantic richness offered by LLMs and the enduring efficiency and effectiveness of the ID-based approach, thereby setting a clear and important agenda for the next generation of recommender system research.