Reducing offline evaluation bias of collaborative filtering algorithms

Recommendation systems have been integrated into the majority of large online systems to filter and rank information according to user profiles. It thus influences the way users interact with the system and, as a consequence, bias the evaluation of the performance of a recommendation algorithm computed using historical data (via offline evaluation). This paper presents a new application of a weighted offline evaluation to reduce this bias for collaborative filtering algorithms.

💡 Research Summary

Recommendation systems are now ubiquitous, but the very act of recommending influences user behavior, which in turn contaminates the historical logs used for offline evaluation. This creates a feedback loop: the algorithm deployed in production shapes the distribution of users and items, causing the joint probability (P_t(u,i)=P_t(u)P_t(i|u)) to drift over time. Consequently, offline scores computed at different moments are not comparable, and algorithms that are actually in production tend to receive inflated offline performance, while orthogonal algorithms are penalized.

The authors formalize this bias and propose a simple yet effective mitigation: re‑weight the conditional item probabilities with item‑specific weights (\omega_i). The new conditional distribution becomes
\