Clear Messages, Ambiguous Audiences: Measuring Interpretability in Political Communication

Text-based measurement in political research often treats classi6ication disagreement as random noise. We examine this assumption using con6idence-weighted human annotations of 5,000 social media messages by U.S. politicians. We 6ind that political communication is generally highly legible, with mean con6idence exceeding 0.99 across message type, partisan bias, and audience classi6ications. However, systematic variation concentrates in the constituency category, which exhibits a 1.79 percentage point penalty in audience classi6ication con6idence. Given the high baseline of agreement, this penalty represents a sharp relative increase in interpretive uncertainty. Within messages, intent remains clear while audience targeting becomes ambiguous. These patterns persist with politician 6ixed effects, suggesting that measurement error in political text is structured by strategic incentives rather than idiosyncratic coder error.

💡 Research Summary

This paper challenges the common assumption in political text analysis that classification disagreements among human coders are merely random noise. Using a dataset of 5,000 social‑media posts (2,500 Twitter, 2,500 Facebook) authored by 505 U.S. federal legislators (both Representatives and Senators), the authors obtain confidence‑weighted annotations across four dimensions: message type, partisan bias, audience (national vs. constituency), and intent. Each coder assigns a confidence score (0–1) to every label, allowing the researchers to compute a confidence‑weighted consensus for each observation.

Descriptive statistics reveal an exceptionally high overall legibility: mean confidence scores exceed 0.99 for message type (0.9962), bias (0.9939), and audience (0.9953). However, a systematic variation emerges when the audience dimension is disaggregated. The “constituency” audience category shows a mean confidence of 0.9834, which is 1.79 percentage points lower than the “national” audience (0.9965). In relative terms, this represents a roughly 180 % increase in interpretive uncertainty compared with the baseline of near‑perfect agreement. Importantly, the intent dimension remains virtually error‑free (confidence ≈ 0.998), indicating that while the purpose of a message is clear, the target audience can be ambiguous.

To test whether this pattern is driven by idiosyncratic coder behavior or by structural features of the messages, the authors estimate mixed‑effects logistic models that include politician fixed effects. The constituency‑audience confidence penalty persists and remains statistically significant (p < 0.01) after controlling for individual legislators, suggesting that the observed uncertainty is not a coding artifact but reflects strategic ambiguity embedded in the messages themselves.

The authors interpret this finding through the lens of political strategy: legislators may deliberately craft constituency‑targeted posts that are vague enough to appeal to multiple sub‑groups within a district, thereby preserving flexibility and reducing the risk of alienating any particular voter bloc. This “strategic ambiguity” creates a measurable, structured source of measurement error that is not captured by conventional reliability checks that focus solely on overall agreement rates.

The paper contributes three main insights. First, it demonstrates that high average confidence does not guarantee uniform interpretability across all analytic dimensions; specific categories can harbor substantial uncertainty. Second, it provides a methodological blueprint for incorporating coder confidence into the aggregation process, offering a more nuanced reliability metric than simple majority voting. Third, it highlights the importance of accounting for strategic communication behavior when interpreting text‑based measures of political behavior.

Limitations include the exclusive focus on U.S. federal legislators, which may limit generalizability to other political systems or to non‑elite political communication. Moreover, confidence scores are still subjective judgments; future work should explore systematic training protocols or automated confidence estimation to reduce coder bias. Extending the approach to multilingual corpora and integrating it with machine‑learning classifiers could further illuminate how strategic ambiguity varies across contexts and platforms.

In sum, the study shows that political communication is generally highly legible, yet audience‑targeting—especially at the constituency level—exhibits a pronounced, strategically driven interpretive uncertainty. Researchers and practitioners should therefore examine category‑specific confidence metrics rather than relying on aggregate reliability, particularly when evaluating the effectiveness of targeted political messaging.