Random Graphs for Performance Evaluation of Recommender Systems

The purpose of this article is to introduce a new analytical framework dedicated to measuring performance of recommender systems. The standard approach is to assess the quality of a system by means of accuracy related statistics. However, the specificity of the environments in which recommender systems are deployed requires to pay much attention to speed and memory requirements of the algorithms. Unfortunately, it is implausible to assess accurately the complexity of various algorithms with formal tools. This can be attributed to the fact that such analyses are usually based on an assumption of dense representation of underlying data structures. Whereas, in real life the algorithms operate on sparse data and are implemented with collections dedicated for them. Therefore, we propose to measure the complexity of recommender systems with artificial datasets that posses real-life properties. We utilize recently developed bipartite graph generator to evaluate how state-of-the-art recommender systems’ behavior is determined and diversified by topological properties of the generated datasets.

💡 Research Summary

The paper addresses a critical gap in the evaluation of recommender systems: while most research focuses on accuracy‑based metrics such as RMSE or Precision@K, real‑world deployments are constrained by execution speed, memory consumption, and scalability. Traditional complexity analyses assume dense data representations, which do not reflect the sparse, high‑dimensional interaction matrices that modern recommender algorithms actually process. To bridge this gap, the authors propose a novel experimental framework that generates artificial bipartite graphs mirroring the structural properties of real user‑item data.

The graph generator is parameterized by several topological characteristics observed in production environments: the total number of users and items, average degree (i.e., average number of interactions per user or per item), the exponent of a power‑law degree distribution, clustering coefficient, and overall connectivity. By adjusting these parameters, the authors can synthesize datasets that emulate a wide range of realistic scenarios—from uniformly sparse interactions to highly skewed “hub‑centric” structures where a few items dominate the interaction space.

Using these synthetic graphs, the study evaluates three families of state‑of‑the‑art recommendation algorithms: (1) memory‑based collaborative filtering (user‑based and item‑based nearest‑neighbor methods), (2) matrix‑factorization techniques (alternating least squares, singular value decomposition), and (3) graph‑neural‑network (GNN) approaches (GCN, GraphSAGE). All algorithms are run on identical graph instances, with consistent hyper‑parameter settings, and their performance is profiled using fine‑grained system monitors that record wall‑clock time, peak RAM usage, and cache‑miss statistics.

Key findings reveal that algorithmic efficiency is highly sensitive to the underlying graph topology. In graphs with higher average degree, memory‑based methods suffer a super‑linear increase in runtime because neighbor list traversals become more expensive and cache locality deteriorates. Matrix‑factorization methods, while relatively stable in memory footprint, experience bandwidth bottlenecks as the number of non‑zero entries grows, leading to longer computation times for dense interaction patterns. GNN‑based recommenders display the most balanced behavior for moderate average degrees, thanks to parallelizable neighborhood aggregation; however, when the power‑law exponent drops below ~2.5 (indicating a few very high‑degree hubs), GNNs also encounter severe memory access irregularities and performance degradation.

The authors also demonstrate that the degree‑distribution exponent (the power‑law α) is a decisive factor: for α ≤ 2.5, all evaluated algorithms exhibit sharply increased memory pressure and cache‑miss rates, making it difficult to meet real‑time latency requirements. This mirrors real‑world phenomena where a small subset of popular items attracts a disproportionate share of user interactions.

Based on these observations, the paper proposes a new evaluation paradigm that explicitly incorporates graph‑topology metrics into complexity analysis. Rather than relying on abstract dense‑matrix assumptions, the framework acknowledges the sparsity and heterogeneity of production data, enabling practitioners to make more informed choices about algorithm selection, hardware provisioning, and system architecture. The authors conclude by outlining future research directions, including automated topology‑aware hyper‑parameter tuning, adaptive recommender models that react to evolving graph structures, and extensions of the synthetic graph generator to incorporate temporal dynamics and side‑information (e.g., content features). This work thus offers a practical, reproducible methodology for assessing the true operational costs of recommender systems in realistic settings.