Evidence of Systematic Bias in 2008 Presidential Polling (preliminary report)

Leonard Adleman Department of Computer Science University of Southern California

Mark Schilling Department of Mathematics California State University, Northridge

Abstract Political polls achieve their results by sampling a small number of potential voters rather than the population as a whole. This leads to “sampling error” which most polling agencies dutifully report. But factors such as nonrepresentative samples, question wording and nonresponse can produce non-sampling errors. While pollsters are aware of such errors, they are difficult to quantify and seldom reported. When a polling agency, whether by intention or not, produces results with non-sampling errors that systematically favor one candidate over another, then that agency’s poll is biased. We analyzed polling data for the (on-going) 2008 Presidential race, and though our methods do not allow us to identify which agencies’ polls are biased, they do provide significant evidence that some agencies’ polls are.

We compared polls produced by major television networks with those produced by Gallup and Rasmussen. We found that, taken as a whole, polls produced by the networks were significantly to the left of those produced by Gallup and Rasmussen.

We used the available data to provide a tentative ordering of the major television networks’ polls from right to left. Our order (right to left) was: FOX, CNN, NBC (which partners with the Wall Street Journal), ABC (which partners with the Washington Post), CBS (which partners with the New York Times). These results appear to comport well with the commonly held informal perceptions of the political leanings of these agencies.

We also compared tracking polls produced by Gallup, Rasmussen, Hotline/FD, and the Daily KOS. Here again we found significant evidence of bias. Most notably, the Rasmussen and the Gallup polls were significantly to the right of the Daily KOS poll. A detailed analysis of the Gallup and Rasmussen polls also suggested the likelihood of short-term bias.

Our findings are preliminary, but given the importance of polling in America, they make a case for further research into the causes of and remedies for polling bias.

Introduction

The influence of opinion polls on political matters has steadily grown in recent years; indeed, poll results now commonly affect political discourse, impact policy decisions and determine campaign strategies. The current election season has spawned a plethora of presidential polls and a number of websites have emerged that combine the information from many polls into a single “state of the race” report that is updated daily.

Unfortunately, polling is not a perfect means of determining the “state of the race”. Polls achieve their results by sampling a small number of potential voters rather than the whole population. This leads to “sampling error”. For most polls, this sampling error ranges from 2-5% and most polling agencies are diligent in reporting this error.

But polls are also subject to non-sampling errors. Pollsters are aware that many factors such as nonrepresentative samples, question wording and question ordering, nonresponse and interviewer bias can affect poll results. The best polling agencies try to mitigate the impact of these factors. When polls are reported in the media, however, non-sampling errors are almost never mentioned. If pollsters were entirely successful in eliminating non-sampling errors, then they could sensibly be ignored by the media and poll consumers. A main purpose of this paper is to provide prima facie evidence that pollsters are not successful. Our analysis shows that in this election cycle some polls have exhibited a left/right political bias.

We collected data from the following sources:

Table 1: Data Sources Data Source Real Clear Politics Average http://www.realclearpolitics.com Rasmussen Tracking Poll http://www.rasmussenreports.com Gallup Tracking Poll http://elections.nytimes.com http://pollingreport.com All other data http://www.realclearpolitics.com

Statistically, bias refers to the tendency for an estimator to produce results that are not centered at the target parameter; that is, the estimates tend to be consistently too high or too low. In presidential polling, however, the target is unknown (except on the day of the election), and it is constantly moving as people’s opinions change over time. In addition, a precise description of the sampling scheme is typically not made public by the polling agency. For these reasons it is not possible for us to determine whether bias is present for an individual agency’s poll. However, it is possible with the data available to investigate whether there is a bias of one agency’s poll relative to another agency’s poll.

To analyze the bias of one agency’s poll relat

…(Full text truncated)…