Nate’s preliminary analysis of pollsters’ performance is out. And the headline is what we’ve all known: Rasmussen is biased. Up to 2009, Scotty’s bias was held in check to some degree. Since then, he’s left it all hang out. And it shows. The proof is in the pudding. (And Nate doesn’t even address Rassmussen’s 12 point final generic).
The 105 polls released in Senate and gubernatorial races by Rasmussen Reports and its subsidiary, Pulse Opinion Research, missed the final margin between the candidates by 5.8 points, a considerably higher figure than that achieved by most other pollsters. Some 13 of its polls missed by 10 or more points, including one in the Hawaii Senate race that missed the final margin between the candidates by 40 points, the largest error ever recorded in a general election in FiveThirtyEight’s database, which includes all polls conducted since 1998.
Moreover, Rasmussen’s polls were quite biased, overestimating the standing of the Republican candidate by almost 4 points on average. In just 12 cases, Rasmussen’s polls overestimated the margin for the Democrat by 3 or more points. But it did so for the Republican candidate in 55 cases – that is, in more than half of the polls that it issued.
If one focused solely on the final poll issued by Rasmussen Reports or Pulse Opinion Research in each state – rather than including all polls within the three-week interval – it would not have made much difference. Their average error would be 5.7 points rather than 5.8, and their average bias 3.8 points rather than 3.9.
Who did well? Q, SUSA (in the last three weeks) and, surpisingly, YouGov:
The most accurate surveys were those issued by Quinnipiac University, which missed the final margin between the candidates by 3.3 points, and which showed little overall bias.
The next-best result was from SurveyUSA, which is among the highest-rated firms in FiveThirtyEight’s pollster rankings: it missed the margin between the candidates by 3.5 points, on average.
SurveyUSA also issued polls in a number of U.S. House races, missing the margin between the candidates by an average of 5.2 points. That is a comparatively good score: individual U.S. House races are generally quite difficult to poll, and the typical poll issued by companies other than SurveyUSA had missed the margin between the candidates by an average of 7.3 points.
In some of the house races that it polled, SurveyUSA’s results had been more Republican-leaning than those of other pollsters. But it turned out that it had the right impression in most of those races – anticipating, for instance, that the Democratic incumbent Jim Oberstar could easily lose his race, as he eventually did.
YouGov, which conducts its surveys through Internet panels, also performed fairly well, missing the eventual margin by 3.5 points on average – although it confined its polling to a handful of swing races, in which polling is generally easier because of high levels of voter engagement.