Improving LLM-based Recommendation with Self-Hard Negatives from Intermediate Layers

Large language models (LLMs) have shown great promise in recommender systems, where supervised fine-tuning (SFT) is commonly used for adaptation. Subsequent studies further introduce preference learning to incorporate negative samples into the training process. However, existing methods rely on sequence-level, offline-generated negatives, making them less discriminative and informative when adapting LLMs to recommendation tasks with large negative item spaces. To address these challenges, we propose ILRec, a novel preference fine-tuning framework for LLM-based recommendation, leveraging self-hard negative signals extracted from intermediate layers to improve preference learning. Specifically, we identify self-hard negative tokens from intermediate layers as fine-grained negative supervision that dynamically reflects the model’s preference learning process. To effectively integrate these signals into training, we design a two-stage framework comprising cross-layer preference optimization and cross-layer preference distillation, enabling the model to jointly discriminate informative negatives and enhance the quality of negative signals from intermediate layers. In addition, we introduce a lightweight collaborative filtering model to assign token-level rewards for negative signals, mitigating the risk of over-penalizing false negatives. Extensive experiments on three datasets demonstrate ILRec’s effectiveness in enhancing the performance of LLM-based recommender systems.

💡 Research Summary

The paper introduces ILRec, a novel fine‑tuning framework that improves large language model (LLM) based sequential recommendation by exploiting self‑hard negative signals extracted from the model’s intermediate layers. Traditional LLM‑Rec approaches rely on a small set of sequence‑level negative items generated offline or by random sampling. Such negatives are coarse, often outdated, and computationally expensive to handle when the candidate item space is huge.

ILRec tackles these issues in three complementary ways. First, it treats the layers just before the final output as “weak experts.” For each training step, hidden states from a selected block of intermediate layers (e.g., L‑2 to L‑4) are passed through an additional prediction head to obtain layer‑specific logits. These logits are averaged to form ensemble logits. Tokens that receive high probability in the ensemble but are not the ground‑truth token are selected as self‑hard negative tokens, using a dynamic threshold τ = α·p̂(y_t) that scales with the confidence on the correct token. This yields a large, token‑level negative set that reflects the model’s current predictive state.

Second, the framework introduces Cross‑Layer Preference Optimization (CPO) and Cross‑Layer Preference Distillation (CPD). CPO augments the standard cross‑entropy loss with a penalty term applied to the final‑layer logits of the selected negative tokens, effectively forcing the model to lower the probability of those tokens. CPD then uses the final layer as a teacher, minimizing the KL‑divergence between the teacher’s distribution and each intermediate layer’s distribution. This distillation step ensures that the intermediate layers generate reliable hard negatives rather than noisy artifacts.

Third, to mitigate the risk of false negatives—tokens that might actually be relevant for a user—a lightweight collaborative filtering (CF) model is trained on user‑item interaction data. The CF model assigns a token‑level reward r(v) ∈