When is it Biased? Assessing the Representativeness of Twitters Streaming API

Twitter has captured the interest of the scientific community not only for its massive user base and content, but also for its openness in sharing its data. Twitter shares a free 1% sample of its tweets through the “Streaming API”, a service that returns a sample of tweets according to a set of parameters set by the researcher. Recently, research has pointed to evidence of bias in the data returned through the Streaming API, raising concern in the integrity of this data service for use in research scenarios. While these results are important, the methodologies proposed in previous work rely on the restrictive and expensive Firehose to find the bias in the Streaming API data. In this work we tackle the problem of finding sample bias without the need for “gold standard” Firehose data. Namely, we focus on finding time periods in the Streaming API data where the trend of a hashtag is significantly different from its trend in the true activity on Twitter. We propose a solution that focuses on using an open data source to find bias in the Streaming API. Finally, we assess the utility of the data source in sparse data situations and for users issuing the same query from different regions.

💡 Research Summary

Twitter’s free Streaming API returns at most 1 % of all public tweets, sampling internally when a query’s volume exceeds that limit. Prior work demonstrated that this sampling can be biased, but verification relied on the costly and restricted Firehose, making routine bias checks impractical for most researchers. This paper proposes a methodology that detects bias in Streaming API data without any Firehose access, using only publicly available resources. The authors introduce Twitter’s Sample API, which provides an unbiased 1 % random sample of the entire firehose without requiring query parameters. They first validate the randomness of the Sample API by collecting tweets containing the keyword “syria” over a four‑hour window and comparing the top‑k hashtag rankings against a Gnip Firehose feed. Kendall’s τβ scores across various k values show strong correlation, and statistical tests reject the hypothesis of independence at the 95 % confidence level, confirming that the Sample API behaves like a random sample of the firehose.
Having established a reliable proxy for the true tweet stream, the authors construct a bias‑detection pipeline. For a given hashtag, they retrieve its time‑series counts from both the Streaming API and the Sample API, then normalize each series (zero‑mean, unit‑variance). They bootstrap the Sample API data 100 times, generating a distribution of normalized counts for each time step. The mean (μᵢ) and standard deviation (σᵢ) of this bootstrap distribution define a 3‑σ control‑chart envelope. If the Streaming API’s normalized count at time i falls outside μᵢ ± 3σᵢ, the point is flagged as biased. The authors illustrate the process with the hashtag #believemovie on August 5, 2013, showing that a spike between hours 10‑11 is under‑represented while later spikes are over‑represented, all statistically significant at the 99.7 % confidence level.
The paper also examines sparsity effects: because the Sample API captures only 1 % of the firehose, low‑frequency (“long‑tail”) hashtags often have many missing observations, limiting bias detection for rare terms, whereas popular hashtags have sufficient data. Finally, the authors test geographic and temporal robustness by issuing identical queries from multiple global locations and overlapping time windows; results are virtually identical, indicating that the Streaming API’s bias is not a function of query origin.
In summary, the study demonstrates that researchers can automatically identify periods of sampling bias in the Streaming API using the free Sample API combined with bootstrap confidence intervals, eliminating the need for expensive Firehose data and providing a practical quality‑control tool for Twitter‑based research.