An Analysis of Chinese Search Engine Filtering

The imposition of government mandates upon Internet search engine operation is a growing area of interest for both computer science and public policy. Users of these search engines often observe evidence of censorship, but the government policies that impose this censorship are not generally public. To better understand these policies, we conducted a set of experiments on major search engines employed by Internet users in China, issuing queries against a variety of different words: some neutral, some with names of important people, some political, and some pornographic. We conducted these queries, in Chinese, against Baidu, Google (including google.cn, before it was terminated), Yahoo!, and Bing. We found remarkably aggressive filtering of pornographic terms, in some cases causing non-pornographic terms which use common characters to also be filtered. We also found that names of prominent activists and organizers as well as top political and military leaders, were also filtered in whole or in part. In some cases, we found search terms which we believe to be “blacklisted”. In these cases, the only results that appeared, for any of them, came from a short “whitelist” of sites owned or controlled directly by the Chinese government. By repeating observations over a long observation period, we also found that the keyword blocking policies of the Great Firewall of China vary over time. While our results don’t offer any fundamental insight into how to defeat or work around Chinese internet censorship, they are still helpful to understand the structure of how censorship duties are shared between the Great Firewall and Chinese search engines.

💡 Research Summary

This paper presents a systematic measurement study of Internet search‑engine censorship in China. Over a 16‑month period (early 2010 to mid‑2011) the authors queried four major search providers—Baidu, Google (global, Google.cn, and Google.hk), Yahoo! (global and Chinese), and Bing (global and cn.Bing.com)—with a total of 45,411 distinct Chinese keywords. The keyword set combines three layers: (1) a large corpus of popular, non‑sensitive terms (the first 44,102 entries from a list of 66,516 common queries), (2) a set of 133 “ConceptDoppler” words previously identified as filtered by the Great Firewall, (3) 1,126 names of Chinese political and military leaders, and (4) 85 ad‑hoc terms added during the experiment to reflect current events.

A custom crawler built on wget simulated a Firefox browser, managed cookies, and sent each keyword both with and without surrounding quotation marks. The returned HTML pages were saved and parsed for the reported number of hits and for the actual result URLs. By comparing hit counts across engines for the same query, the authors inferred the presence and degree of censorship.

Key findings include:

• Pornography filtering is extremely aggressive. Not only are explicit porn terms blocked, but any phrase sharing the same characters can be inadvertently filtered, demonstrating character‑level over‑filtering.
• Political and activist terms (e.g., Falun Gong, Tiananmen‑related phrases, names of dissidents) are either fully suppressed or partially reduced. In many cases the only results returned belong to a short “whitelist” of government‑controlled sites, indicating a deliberate “blacklist/whitelist” strategy.
• Leader name censorship shows a spectrum: some high‑ranking officials are completely omitted, while others appear with reduced hit counts, suggesting tiered sensitivity.
• Temporal dynamics are evident. Certain keywords fluctuate between blocked and unblocked states over weeks, reflecting policy updates or adaptive filtering rules.
• Engine‑specific behavior varies. Baidu exhibits the strongest self‑censorship, often returning only government‑approved pages. Google, even after the shutdown of Google.cn, still applies keyword filters but is more transparent about result reduction. Yahoo! and Bing display lighter filtering but still enforce a whitelist for highly sensitive queries.
• Quotation marks and character substitution (e.g., replacing a radical with a visually similar one) can bypass some filters, confirming that the censoring mechanisms rely heavily on exact string matching.

The study concludes that Chinese Internet censorship is not solely the product of the national firewall; search providers embed their own filtering logic, maintain keyword blacklists, and serve curated whitelists. The authors highlight the importance of continuous measurement, note the limitations of their methodology (e.g., reliance on hit counts, possible cache effects), and suggest future work on real‑time detection of keyword mutations and on advocating greater algorithmic transparency from search companies.