Popularity and Quality in Social News Aggregators: A Study of Reddit and Hacker News

In this paper we seek to understand the relationship between the online popularity of an article and its intrinsic quality. Prior experimental work suggests that the relationship between quality and popularity can be very distorted due to factors like social influence bias and inequality in visibility. We conduct a study of popularity on two different social news aggregators, Reddit and Hacker News. We define quality as the relative number of votes an article would have received if each article was shown, in a bias-free way, to an equal number of users. We propose a simple poisson regression method to estimate this quality metric from time-series voting data. We validate our methods on data from Reddit and Hacker News, as well the experimental data from prior work. This method works well even though the collected data is subject to common social media biases. Using these estimates, we find that popularity on Reddit and Hacker News is a stronger reflection of intrinsic quality than expected.

💡 Research Summary

The paper investigates how well the observed popularity of articles on two major social news aggregators—Reddit and Hacker News—reflects their intrinsic quality. Prior experimental work, especially the MusicLab study by Salganik, Dodds, and Watts, demonstrated that popularity can be heavily distorted by rich‑get‑richer dynamics and social influence, leading to a weak correlation between true quality and observed success. Building on this insight, the authors treat Reddit and Hacker News as ideal testbeds because their ranking interfaces are relatively simple, non‑personalized lists where position bias can be measured similarly to search result bias.

Data were collected over a two‑week period (May 26 – June 6 2014) at ten‑minute intervals. For Hacker News, the authors scraped both the “new” and “top” rankings, recording up‑votes, comments, and the position of each story. For Reddit, they focused on the hot rankings of several default subreddits, capturing up‑votes, down‑votes, scores, and positions. Because Reddit applies “vote fuzzing” to mitigate spam, raw vote counts were transformed using a method described in the appendix.

The core methodological contribution is a Poisson regression model that estimates a latent “quality” parameter for each article while explicitly accounting for position bias, temporal decay, and potential social influence. The baseline model assumes the expected number of votes λ for article i observed at time t in position j follows λ_{tij}=exp(α_i+β_j), where α_i captures the intrinsic quality of the article and β_j captures the visibility advantage of position j. The authors extend this model by adding a time‑decay term (γ·age) and a term for the current cumulative vote count (δ·cum_votes) to model how social signals may amplify voting probability over time.

Parameters are estimated via maximum likelihood across all observed (t,i,j) tuples. The resulting α_i values are interpreted as the number of votes an article would receive if it were shown to the same number of users under a bias‑free condition—i.e., the “intrinsic quality” score. To validate the approach, the same Poisson framework is applied to the MusicLab dataset, where true quality is known from the control world with random ranking. The model recovers quality parameters with a correlation of about 0.78 to the ground truth, confirming its ability to disentangle quality from popularity‑inducing biases.

Applying the model to Reddit and Hacker News yields several key findings. First, a strong position bias is confirmed: on Hacker News, stories that initially appear on the front page (positions 1‑30) achieve on average 57 more up‑votes than those starting on the second page, even after controlling for submission time and day. This demonstrates that early exposure dramatically shapes eventual success. Second, despite this bias, the estimated quality parameters α_i correlate positively and substantially (≈0.6–0.68) with final popularity scores, indicating that the most popular items are indeed among the highest‑quality ones in the system. In other words, the “digital democracy” of these platforms is not wholly broken; when visibility is accounted for, quality does translate into popularity.

The paper also revisits prior work on Reddit reposting. By analyzing the number of times the same external URL is submitted across different subreddits, the authors find a positive relationship between repost frequency and external web popularity. Multiple submissions increase the chance that at least one will land in a high‑visibility position, thereby boosting overall visibility and votes. This suggests that while reposting may appear as gaming the system, it can also serve as a mechanism that eventually surfaces high‑quality content.

In summary, the study makes three major contributions: (1) it proposes a simple yet effective Poisson regression framework for estimating intrinsic article quality from observational voting data; (2) it empirically demonstrates the magnitude of position bias and the importance of correcting for it; and (3) it shows that, after such correction, popularity on Reddit and Hacker News aligns closely with intrinsic quality. These insights have practical implications for the design of ranking algorithms and for understanding how social platforms can better promote high‑quality information without sacrificing user engagement.