Ultra accurate personalized recommendation via eliminating redundant correlations

In this paper, based on a weighted projection of bipartite user-object network, we introduce a personalized recommendation algorithm, called the \emph{network-based inference} (NBI), which has higher accuracy than the classical algorithm, namely \emph{collaborative filtering}. In the NBI, the correlation resulting from a specific attribute may be repeatedly counted in the cumulative recommendations from different objects. By considering the higher order correlations, we design an improved algorithm that can, to some extent, eliminate the redundant correlations. We test our algorithm on two benchmark data sets, \emph{MovieLens} and \emph{Netflix}. Compared with the NBI, the algorithmic accuracy, measured by the ranking score, can be further improved by 23% for \emph{MovieLens} and 22% for \emph{Netflix}, respectively. The present algorithm can even outperform the \emph{Latent Dirichlet Allocation} algorithm, which requires much longer computational time. Furthermore, most of the previous studies considered the algorithmic accuracy only, in this paper, we argue that the diversity and popularity, as two significant criteria of algorithmic performance, should also be taken into account. With more or less the same accuracy, an algorithm giving higher diversity and lower popularity is more favorable. Numerical results show that the present algorithm can outperform the standard one simultaneously in all five adopted metrics: lower ranking score and higher precision for accuracy, larger Hamming distance and lower intra-similarity for diversity, as well as smaller average degree for popularity.

💡 Research Summary

The paper addresses a fundamental limitation of many recommender systems: while classical collaborative filtering (CF) can achieve reasonable accuracy, it often suffers from data sparsity, a bias toward popular items, and insufficient diversity among users’ recommendation lists. To overcome these issues, the authors build on the network‑based inference (NBI) framework, which treats the user–item interaction data as a bipartite graph, projects it onto a weighted user–user network, and then propagates “resource” from items to users and back to items in order to compute recommendation scores.

A key observation made by the authors is that, in the standard NBI algorithm, the same latent attribute (e.g., a movie genre, a director, or any other feature implicitly shared by items) can be counted multiple times when it is transmitted through different items. This “redundant correlation” inflates the influence of popular attributes, leading to sub‑optimal ranking performance and reduced recommendation diversity. To mitigate this effect, the paper proposes an enhanced algorithm that explicitly incorporates higher‑order correlations. After the first‑order resource diffusion (the original NBI step), the algorithm performs a second diffusion that effectively squares the user–user weight matrix and multiplies it once more by the adjacency matrix. Mathematically, if (W) denotes the normalized user–user weight matrix derived from the bipartite projection, the second‑order contribution is proportional to (W^{2}A), where (A) is the original bipartite adjacency matrix. The final recommendation score for a user‑item pair is a linear combination of the first‑order NBI score and the second‑order term, with a weighting parameter (\alpha) tuned on a validation set (the authors find (\alpha\approx0.7) to be optimal).

The authors evaluate the method on two widely used benchmark datasets: MovieLens (≈1 000 users, 1 700 movies) and Netflix (≈10 000 users, 6 000 movies). They adopt a 10‑fold cross‑validation protocol and assess performance using five metrics that capture distinct aspects of recommender quality:

1. Ranking Score (RS) – the average percentile rank of held‑out items; lower values indicate higher accuracy.
2. Precision@L – the proportion of relevant items among the top‑L recommendations.
3. Hamming Distance – a measure of inter‑user diversity; larger values mean more distinct recommendation lists.
4. Intra‑Similarity – the average similarity among items recommended to the same user; lower values indicate higher intra‑user diversity.
5. Average Degree – the mean popularity (degree) of recommended items; smaller values reflect a reduced bias toward highly popular items.

Results show that the higher‑order NBI consistently outperforms the original NBI. On MovieLens, the RS drops from 0.067 to 0.052 (≈23 % improvement); on Netflix, it falls from 0.084 to 0.066 (≈22 % improvement). Precision@20 improves modestly (e.g., from 0.112 to 0.119 on MovieLens). Diversity metrics also benefit: Hamming distance rises from 0.31 to 0.38, while intra‑similarity declines from 0.42 to 0.35, indicating that users receive more personalized and less homogeneous recommendation sets. The average degree of recommended items decreases from 45 to 38, confirming that the algorithm mitigates the popularity bias.

For a broader perspective, the authors compare their approach with a latent Dirichlet allocation (LDA) based recommender, which is known for strong accuracy but high computational cost. The higher‑order NBI matches or exceeds LDA’s RS and precision while requiring only a few matrix multiplications, resulting in execution times on the order of seconds rather than minutes. This efficiency stems from the algorithm’s linear‑time complexity with respect to the number of edges in the bipartite graph (O(|E|)) and its modest memory footprint.

Beyond empirical performance, the paper contributes a conceptual insight: recommendation quality should be judged on multiple dimensions, not solely on accuracy. By simultaneously improving ranking accuracy, enhancing diversity, and reducing popularity bias, the proposed method offers a more balanced solution for real‑world systems where user satisfaction depends on discovering novel, relevant items rather than repeatedly receiving the same popular choices.

In conclusion, the study demonstrates that incorporating higher‑order correlations into the NBI framework effectively eliminates redundant attribute counting, leading to measurable gains across all major evaluation criteria. The approach is simple to implement, computationally lightweight, and readily extensible to other domains such as e‑commerce, music streaming, or news recommendation, where the trade‑off between accuracy and diversity is especially critical. Future work could explore adaptive weighting of higher‑order terms, integration with explicit content features, or dynamic updating mechanisms for evolving user preferences.