February 12, 2008

Why The Polls Can't Be Trusted

The 2008 New Hampshire Presidential Primaries were notable in that the actual results of the Democratic primary differed greatly from pre-election polls. Here's a breakdown of a few pre-election poll numbers, with the actual results of the Primary listed below:

We all know about Hillary’s 11th hour “teary-eyed” appeal, you know, where she lamented how this election is “very personal” and “not just political,” and that “some people think elections are a game.” And then, just as we think the former First Lady has cast aside petty partisan discourse, even for a minute, she pulls a fast one and rips into a subtle but acute criticism of her chief rival, arguing:

“but some of us are right and some of us are wrong. Some of us are ready and some of us are not. Some of us know what we will do on day one, and some of us have not really thought that through well enough.”

But I digress…

Most pundits declared this the principal factor in Hillary’s triumph in New Hampshire. Given my desire to avoid repeating the obvious, I’d like to explore a couple other reasons why the pre-election polls erred so egregiously:

1) Selection Bias due to the demographic gap between supporters of Clinton and Obama:

A principal reason for the discrepancy is a selection bias that resulted from the demographic gap (in terms of education levels) between Clinton supporters and Obama supporters. CNN exit polls in New Hampshire showed that 43% of postgraduates supported Obama, compared to only 31% for Clinton. Among high school graduates, 46% supported Clinton compared to 31% for Obama. Clinton carried the vote among those without a high school diploma by an astonishing 61% to 25% margin.

Evidence also shows that less-affluent citizens are much less likely to participate in surveys and polls than affluent citizens. Since Obama earns the bulk of his support from the more affluent (as demonstrated above), it is very likely that a larger proportion of Obama supporters participated in pre-election polls than did Clinton supporters, thus skewing the poll results. This phenomenon is what researchers refer to as a non-response error, or selection bias.

2) Defection of Independent Voters to McCain:

Next, by heavily favoring Obama, the pre-election polls may have worked to his disadvantage among independents in New Hampshire, who can vote in either the Democratic or the Republican primary. Obama earned ample support in his Iowa Caucus victory among registered independents, with as many as 40% of registered independents jumping on his bandwagon. Independents in New Hampshire, seeing that pre-election polls showed Obama with an 11% lead, may have decided to swing to the Republican side and cast their vote for McCain, another favorite among independents.

Pre-election polls showed that the Republican race was very close and that the Democratic race was not. Independents, who tend to respond more favorably to Obama and McCain, may have decided that McCain needed their vote more than Obama. Thus, I posit that while these independent voters supported Obama in pre-election polls, they swung their vote to another well-liked candidate (McCain) perceived to be in greater need of their independent votes. It was a strategic vote, and it ended up costing the Illinois Senator in the end.

So what am I getting at? The next time you glance at the polls and think they’re reliable, think again.