American politics does not suffer from a shortage of polls. Zogby. Gallup. Rasmussen. SurveyUSA. Mason-Dixon. Polimetrix…- In an information-glutted world, what matters is not the supply of sources, but the ability to glean trustworthy information from the larger swath of poor data.
Different polling organizations have different strengths and weaknesses. Some use “-tight screens”- to scope out likely voters- others simply sample registered voters, without making any attempt to tighten the survey base to “-likely voters.”- Tight screening is especially crucial to gauge the true state of a primary, when committed base opinion can diverge significantly from less engaged moderate voters, and more importantly, influence those moderates over time to converge to the more partisan perspective. Some use human interviewers, although recently that has given way to IVR (Interactive Voice Recording) polls (the kind where a computer talks to you and asks you to “-press 1 if you will definitely support X, 2 if probably…-”-)
I have found tight-screen, IVR polling to be the most reliable. IVR not only has no marginal cost, but it eliminates all the biases resulting from trying to give the most pleasant-sounding answer possible (the “-sexy grad student effect”- that exaggerated Kerry’-s margin by 15 points in Pennsylvania 2004 exit polling, for example). IVR possible responses can also be randomly rotated from respondent to respondent to eliminate recency biases (first and last responses in a list exaggerated because those are at the forefront of a person’-s memory of the list, not because s/he will vote that way).
The poster-child of IVR tight-screen polling success is Scott Rasmussen’-s Rasmussen Reports. I have only tracked them over the last two election cycles (2004 and 2006), but considering that 2004 was a GOP wave and 2006 a Democratic wave election, I think the data is sufficient to form a valid judgment. Rasmussen’-s track record is simply stupendous. It predicted 49 out of 50 states in 2004 correctly, usually within two percentage points of the actual outcome. In 2006, Rasmussen achieved similarly impressive results —- all the more impressive when you consider that most polling models tend to err in favor of one party or the other. (“-Likely voter”- models tend to favor Republicans, and registered voter-based models tend to exaggerate Democratic strength.)
My other favorite sources include Gallup and Mason-Dixon. Gallup comes closer to the “-registered voter”- model than the tighter Rasmussen model, so Gallup usually lags tighter-screen polls. By election eve, however, the two models usually converge. Gallup’-s election-eve congressional generic vote is hands-down the best in the business. However, their numbers for party primaries have poor predictive value, because they don’-t make much effort to hunt down likely voters.
Differing survey methods can yield very different results. Rasmussen has long shown a much closer Democratic nomination race than most established, “-registered voter”- pollsters —- most recently, it showed a 32-32 tie between Clinton and Obama, with Edwards wallowing 15 points behind. Gallup’-s last numbers tightened drastically to a 31-26 race between Clinton and Obama (Gallup’-s numbers are also hard to compare with Rasmussen’-s because Gallup includes Gore).
Many smart Democrats, notably MyDD’-s Chris Bowers, believe that Gallup and others are mistakenly including lots of “-low information voters”- who simply lag the opinions and thought processes of more-attuned Democratic partisans.
Now that more establishmentarian polling firms are coming in line with Rasmussen’-s results, one can infer that the likely voter/ Chris Bowers theory has gotten the better of the argument.
A survey of pollsters wouldn’-t be complete without knowing which ones to stay away from. Stay away from Zogby and CNN polling. James Carville’-s and Stan Greenberg’-s DemocracyCorps polling outfit is not trustworthy, either —- for example, when they doubled the percentage of blacks in an October 2006 survey sample to bump the Democrats’- generic advantage by 5 points, to reinforce the Democratic narrative of a building wave.
Lastly, partisan pollsters in a competitive election season should always be taken with a grain of salt —- they will use heuristic subtleties to create the best impression possible for their party’-s candidates. Strategic Vision, a Republican outfit, deserves a three- or four-point handicap. Franklin Pierce generated a dubious Romney result for New Hampshire right after its lead pollster, Rich Killion, went to work for the Romney campaign. Such polls should be trusted only as a last resort.
For those of us who wish to divine movements in politics futures, discerning trustworthy data from bad data is paramount. Poll-rigging is the high art of Washington, DC, and as any interest group —- or candidate —- knows, it’-s easier than easy to produce a poll that diverges wildly from reality, if the heuristics are threatening enough.
(cross-posted from my blog, The Tradesports Political Maven)
