Polling the Pollsters

With the outsider and/or netroots candidates (I threw in the “and/or” because Jack Conway doesn’t really fit the “outsider” tag) having overperformed polling expectations in basically all the major primaries (Arkansas, Pennsylvania, and both sides in Kentucky), I thought it would be worth a quick look at what pollsters came closest to getting it right before Tuesday.

• Arkansas: The two polls in the two weeks, from R2K and Mason-Dixon before the primary both gave Blanche Lincoln a lead in the 10-ish ballpark, quite different from her 45-42 finish. The closest prediction came from back in mid-April, in the TalkBusiness poll (conducted by Zeta|3), which gave Lincoln a 7-point lead (38-31) and was also the only pollster to see D.C. Morrison breaking into the double digits (at 10 — he ended up at 13).

• Kentucky (D): The night’s closest result, a 44-43 win for Jack Conway, has to be seen as a surprise: no public pollster had given Conway a lead over Dan Mongiardo since PPP back in December. (A Conway internal in early May gave himself a 4-point lead.) Honors in this category probably go to SurveyUSA, who one week before the primary foresaw a dead heat, in the form of a 38-37 Mongiardo lead.

• Kentucky (R): The 59-35 Randslide was a bigger Paul edge than most pollsters saw, although if you follow the trendlines, there was a pretty clear break for Paul among undecideds at the end that put him on a trajectory for 59. The three last pollsters to hit this race all caught that trend: SurveyUSA on the 9th-11th saw it at 49-33, and PPP and Magellan on the 15th-16th saw 52-34 and 55-30 respectively. I’ll give Republican firm Magellan the nod, for almost nailing the margin.

• Pennsylvania: As I’d expected, Pennsylvania’s primary saw the incumbent rule making a comeback, as most undecideds broke Joe Sestak’s way for a 54-46 victory over Arlen Specter (the Pollster aggregate going in was 43-all). With most pollsters projecting a dead heat, I was a little skeptical at the time of Suffolk’s one-week-out prediction of a 9-point margin (49-40), but that turned out very close to the actual margin.

• PA-12: There’s not much polling data to compare here (Pollster doesn’t have an aggregate), but two of the pollsters that we tend to like, PPP and R2K, both whiffed, predicting a narrow Tim Burns victory (rather than the actual 53-45 Mark Critz win). The winner here is the Global Strategy Group poll that came straight from the Critz camp, accurately prediciting an 8-point spread (44-36); the best public poll came from, of all places, Susquehanna (a Pennsylvania-based Republican pollster, although working for the Pittsburgh Tribune-Review rather than Burns here), whose last poll of the race gave a 44-38 edge to Critz.

While we’re talking about the 12th, how did we at SSP do? We had a sophisticated model running behind the scenes helping us with our projections as the night progressed, and our numbers guru jeffmd shares his assessment:

The model (in my not-so-unbiased opinion) did decently well, but as with these types of models, suffers from the all-too-tenuous uniform swing assumption. Uniform swing was clearly not out in force today, considering, for example, a 6.63% Critz overperformance in Cambria County (Critz’s base) and a 5.83% underperformance in Washington County (Burns’ home base). Variability in precinct size is also an issue. In terms of predicting percentages, the model got pretty close with about 1/3 of precincts in (see graph below), but in terms of raw votes, the model seemed to underestimate the number of votes cast (second graph). For what it’s worth, this is a problem also present in the Coakley-Brown model. Regardless, it’s something SSP Labs will continue to refine moving forward.