So How’d We Do?

During the election season, people seemed to assume that Swing State Project has long been in the business of handicapping congressional races. However, believe it or not, 2008 was the first time that SSP attempted to rate and predict every congressional race. It was an extremely fun project, as we argued back and forth, trying to sell each other on particular candidates’ hidden strengths or districts’ unique quirks, parsing the meaning of “Lean” and “Likely,” or simply trash-talking each other. (In order to briefly return to those golden days, this is a fully collaborative post, and David and James have their say further down too.)

Now that every House race has finally been called and things have settled back down to business as usual here, we thought we’d do a little retrospective and see how our predictions matched up with the actual results. (Our final predictions are here.) Our table is broken into races where Team Blue was on the offense and on the defense, ordered in terms of the margin of victory (or loss). (An asterisk refers to a race that was once on the chart, but dropped off by the end.) Even if you aren’t that interested in our slightly belated soul-searching about our predictive skills, this should be a very useful chart for our readers, as the decreasing margins give a pretty clear picture of who’s vulnerable going into the next cycle.

District Offense Margin Rating District Defense Margin Rating
NY-13 Open 27.6 Safe D FL-16 Mahoney -20.2 Likely R
IL-11 Open 23.9 Lean D LA-06 Cazayoux -7.8 Tossup
AZ-01 Open 20.5 Likely D TX-22 Lampson -7.0 Tossup
FL-24 Feeney 16.1 Lean D KS-02 Boyda -4.4 Lean D
NY-25 Open 12.9 Likely D LA-02 Jefferson -2.7 Safe D
CO-04 Musgrave 12.4 Lean D PA-11 Kanjorski 3.2 Tossup
NM-02 Open 12.0 Tossup AL-05 Open 3.6 Lean D
VA-11 Open 11.7 Lean D NY-24 Arcuri 4.0 Safe D *
NM-01 Open 11.4 Lean D NH-01 Shea-Porter 5.9 Lean D
NC-08 Hayes 10.8 Lean D TX-17 Edwards 7.5 Safe D
OH-16 Open 10.8 Lean D WI-08 Kagen 8.1 Lean D
MI-09 Knollenberg 9.5 Lean D FL-22 Klein 9.4 Safe D
NV-03 Porter 5.1 Tossup AZ-05 Mitchell 9.6 Lean D
OH-01 Chabot 4.9 Tossup ME-01 Open 9.8 Safe D
VA-02 Drake 4.9 Lean R CA-11 McNerney 11.6 Lean D
NJ-03 Open 4.2 Tossup MS-01 Childers 10.6 Likely D
FL-08 Keller 4.0 Tossup PA-04 Altmire 11.8 Likely D
CT-04 Shays 3.7 Tossup AZ-08 Giffords 11.9 Likely D
PA-03 English 2.4 Tossup PA-10 Carney 12.6 Lean D
MI-07 Walberg 2.3 Tossup TX-23 Rodriguez 13.9 Likely D
NY-29 Kuhl 2.0 Lean D IA-03 Boswell 14.3 Safe D
ID-01 Sali 1.2 Tossup GA-08 Marshall 14.4 Lean D
MD-01 Open 0.8 Tossup NH-02 Hodes 15.0 Safe D *
OH-15 Open 0.7 Lean D PA-08 P. Murphy 15.2 Likely D
AL-02 Open 0.6 Lean R IL-14 Foster 15.4 Likely D
VA-05 Goode 0.2 Lean R PA-12 Murtha 15.8 Lean D
LA-04 Open -0.4 Tossup OR-05 Open 16.0 Likely D
CA-04 Open -0.6 Tossup KS-03 Moore 16.8 Likely D
CA-44 Calvert -2.4 Safe R NY-01 Bishop 16.8 Safe D
MO-09 Open -2.5 Tossup NY-19 Hall 17.4 Safe D *
MN-06 Bachmann -3.0 Tossup OH-10 Kucinich 17.9 Safe D
NE-02 Terry -3.8 Tossup IA-02 Loebsack 18.4 Safe D
SC-01 Brown -4.0 Lean R KY-03 Yarmuth 18.8 Likely D
PA-06 Gerlach -4.2 RTW * PA-07 Sestak 19.2 Safe D
CA-50 Bilbray -5.1 Likely R IN-09 Hill 19.4 Likely D
AK-AL Young -5.2 Lean D TX-27 Ortiz 19.5 Safe D
IL-10 Kirk -5.2 Tossup OH-18 Space 19.8 Safe D *
KY-02 Open -5.2 Lean R CT-05 C. Murphy 20.2 Likely D
CA-03 Lungren -5.5 RTW TN-04 Davis 21.0 Safe D
WA-08 Reichert -5.6 Tossup IL-08 Bean 21.4 Safe D *
MI-11 McCotter -6.0 Safe R WI-07 Obey 21.7 Safe D
FL-25 M. Diaz-Balart -6.2 Tossup CO-03 Salazar 23.2 Safe D
OH-02 Schmidt -7.2 Lean R FL-02 Boyd 23.8 Safe D
SC-02 Wilson -7.5 RTW ND-AL Pomeroy 24.0 Safe D
MN-03 Open -7.6 Tossup NY-20 Gillibrand 24.2 Lean D
NJ-07 Open -8.0 Tossup SC-05 Spratt 24.6 Safe D
AL-03 Rogers -8.2 Likely R WA-02 Larsen 24.8 Safe D
CA-46 Rohrabacher -9.5 Likely R NM-03 Open 26.2 Safe D
WY-AL Open -9.8 Lean R NC-11 Shuler 26.2 Safe D *
IL-13 Biggert -9.9 RTW NC-04 Price 26.6 Safe D

As you can see, by the time you get up to 50, the Democratic defense list has started to get kind of uninteresting, while there are still some hotly contested offense seats left to discuss. It’s a pretty good illustration of how lopsided the playing field for the two parties was this year. For instance, there’s only one Democratic defense seat that we had left on our big board that fell off the list: Tim Walz in MN-01, who was Likely D but won by 29.6% (good for 67th place).

On the other hand, here’s the continued list for offense seats!

51) NV-02, Heller, -10.4, Lean R

52) TX-10, McCaul, -10.8, Lean R

54) AZ-03, Shadegg, -11.1, Lean R

57) NJ-05, Garrett, -13.5, Lean R

58) TX-07, Culberson, -13.5, Likely R

59) WV-02, Capito, -14.2, Lean R

60) NY-26, Open, -14.5, Lean R

67) NC-10, McHenry, -15.2, Likely R

68) IN-03, Souder, -15.3, Tossup

72) FL-18, Ros-Lehtinen, -15.8, Likely R

73) FL-21, L. Diaz-Balart, -15.8, Tossup

77) OH-07, Open, -16.4, Likely R

80) NC-05, Foxx, -16.8, Likely R

81) PA-15, Dent, -17.2, Likely R

83) FL-13, Buchanan, -18.0, Likely R

89) VA-10, Wolf, -20.0, Likely R

95) IA-04, Latham, -21.2, Likely R

103) MO-06, Graves, -22.5, Likely R

128) LA-07, Boustany, -27.6, Likely R

142) LA-01, Scalise, -31.4, Likely R

The first thing I notice is that there are only six places where we got it “wrong,” where wrong means we felt that, rather than leaving a race as “Tossup,” we could move it to “Lean” or even “Likely…” only to see it go the wrong way. On the defense side, that means Bill Jefferson at Safe D, whose loss I think absolutely no one saw coming (the NRCC’s four-digit campaign expenditures notwithstanding). It also means Nancy Boyda at Lean D. Although she seemed to have a comfortable edge in polls, her surprise loss provides a nice object lesson for incumbents defending tough districts: don’t try to run a campaign that actually appeals to your constituents’ logic and good judgment. Accept the DCCC’s money, and use it to run negative TV spots, instead of trying to engage them intellectually with policy-specific newspaper ads.

On the offense side, the big screwup is Don Young at Lean D; again, this is one that basically no pundit saw coming, thanks to extremely consistent polling in favor of Ethan Berkowitz. The lesson here: never underestimate Alaskans’ willingness to vote for more pork, even if it means supporting a felon (or soon-to-be felon) in the privacy of the voting booth.

We also had something of a crisis of faith in Bobby Bright in AL-02, in the face of tepid campaigning and a crimson district. Despite our dropping him late in the game to Lean R, his name rec and DCCC spending seemed to pull him over the line. Finally, we were caught off guard by the magnitude of the Obama coattails in Virginia, where we left Glenn Nye (VA-02) and Tom Perriello (VA-05) at Lean R. The polls just weren’t there for them, in GOP-leaning turf, but the bluening of Virginia lifted them far enough. (If there’s one candidate I’m personally shocked that won, it’s Perriello; I was miffed to see the DCCC pouring money into a guy who seemed way too progressive for such a rural and downscale district. Here’s one race where I’m super-happy to eat some crow.)

Where else did we whiff? IN-03 and FL-21 seemed like Tossups at the time, given the very close polling and baffled-seeming incumbents, but these ones are languishing up around #70. Apparently the constituents decided late in the game that, in IN-03, they had a challenger they just didn’t know enough about (Mike Montagano), and in FL-21, probably a challenger that they just knew too much about (Raul Martinez).

We may also have been a little generous on the Louisiana challenges in LA-07 and LA-01 (both listed as Likely R). Jim Harlan, with a conservative profile and his own fat pocketbook, seemed like the best possible candidate for LA-01; however, given that this is one of the nation’s most right-wing districts, I guess we have to take a 30-point loss (instead of the usual 50-point beatdown that we take in that district) as some sort of moral victory.

On the flipside, we missed a number of strong performances in California, especially the near upsets of Ken Calvert in CA-44 and Dan Lungren in CA-03. What’s most interesting is that the rising blue tide in California seemed very evenly distributed throughout the state and probably tied to an Obama-driven boost in infrequent voters voting straight-ticket D, as higher-profile challenges to Dana Rohrabacher and David Dreier did only slightly better than completely under-the-radar challenges to guys like Buck McKeon, Wally Herger, and Elton Gallegly.

Where did we buck the odds? I’m pleased with how well we did at moving the right people to “Lean D” in the weeks before the election; at the time, it seemed a little audacious to call a win in advance for Gary Peters, Larry Kissell, Suzanne Kosmas, Betsy Markey, and Eric Massa in their fights against (lame) incumbents, but they all pulled it out… as did last-minute change Mary Jo Kilroy, who finally managed to pull it out in overtime and save us a lot of egg on our faces.

On the whole, we ran up a pretty good track record (while using the ass-covering category of “Tossup” a lot less than certain other prognosticators). The lesson here is that prognosticating is more art than science; your predictions are only as good as your polls and your scuttlebutt.

DavidNYC: This was indeed a very fun project and a tremendous learning experience, and I expect will continue to do race ratings in the future. It was also remarkably time-consuming, especially as we got toward the end – as Crisitunity suggests, there was a lot of back-and-forth as we pored over Google spreadsheets – plus the occasional bit of smack talk. But I think we’d all gladly take more cycles like the one just concluded!

I just have a few additional thoughts. I think our Senate ratings hit the mark, and I think we were in general pretty disciplined in not moving races until we had sufficient evidence to justify a change. Some examples I’m thinking of include OR-Sen and CT-04, where we insisted on seeing polling before concluding that popular, “moderate” Republican incumbents were truly in jeopardy.

On the flipside, I think sometimes you just have to acknowledge an open seat is gone, as we did early on by moving VA-Sen to Safe D in August, and later NY-13 in October. (Both of these were thirty-point races.)

One of our biggest flubs, though, was NY-24. We had the race as Likely D until a week before election day, when we moved it to Safe. A lack of polling, zero outside spending, and a seemingly unimpressive Republican who had been substantially outraised all convinced us that there was nothing to see here.

We couldn’t have been more wrong. In the end, Mike Arcuri raised “just” $1.6 million (unimpressive compared to fellow freshmen like Chris Murphy, Patrick Murphy or even Paul Hodes), while Richard Hanna took in almost $1.1 mil. The final four-point margin was hair-raising, and suggests Arcuri still has a lot of work to do to establish himself. It also tells us that there will always be surprises – and that absence of evidence is not evidence of a Straniere.

James: Crisitunity and David touched on a lot of key points above, but I’ll just add that I think that we all were a bit caught off guard by just how much of a focal point our race ratings exercise became in the day-to-day operations of this blog.

When I first drafted a preliminary set of race ratings at the tail end of 2007, David’s response after I asked him for his thoughts was merely: “Nice work!” David later admitted to me that he felt as if he were a busy parent being handed a crappy piece of crayon art by a proud six year-old son. But once that crayon drawing was slapped on the fridge, if you will, we all realized that we would have to put in a great level of care into making sure we felt that each rating had a strong leg to stand on. In that sense, our race ratings project became the engine of SSP: we all had to step up our game to make sure that no major (or even minor) developments in the key House races would slip past us unnoticed — especially after we achieved some early success by noticing MS-01 before anyone else did.

Our goal of making this ratings project as honest and credible as possible, I believe, had a great impact on our front page coverage, and I know we caught on to a lot of stories and developments that we may have otherwise missed had it not been for our relentless commitment to stay on top of things. There’s no doubt in my mind that our ratings exercise, even if it provided no great revelations to anyone else, helped improve the work and quality of this blog immensely over my output in the summer and fall of 2006.

27 thoughts on “So How’d We Do?”

  1. It’s not easy to guess these things even in the districts you think you know well.

    I wasn’t following the races in other states nearly as closely, but I thought IA-04 should have been “lean R”. Latham’s 21-point win clearly indicates that you were right to call it likely R.  

  2. My father told me that years before he always worried about the weather on election day.  He told me that although statewide elections may not be as affected by the weather (in NC, you can have warm weather in the Piedmont, rain in the mountains, and heavy rains on the East Coast), but the local elections can be influenced more, especially if the elderly and the poor cannot get to the polls.

    I imagine a lot of this is mitigated by early voting, but I thought I’d throw that out for everyone to consider.  It sounds like this argument has some logic to it.

    I think you did a great job with your predictions!

  3. I appreciate all the hard work you guys did in bringing us the race ratings.  It certainly made following the races more interesting and enjoyable.

    Finally, a mea culpa.  I recall chastising James for not moving Berkowitz from toss up to lean D some few weeks before the election.  You never should’ve listened to me!

  4. I too did the best I could with my predictions, being as objective as I could. I was a bit surprised by Bobby Bright’s win in AL-02 though I changed that race to a Dem win at the last minute. I was also surprised at how the Alaska results differed wildly from my predictions. I guess I may have to include a gut feeling factor in some of these races such as AK-AL. Fortunately, I may not have to do that for the Senate race with the King of Pork outta there!

  5. And since we are reviewing rankings I just have to take the opportunity to gloat about Perriello. I don’t think anyone else publicly predicted he would win. That was a pretty awesome highlight of election night.

  6. That was fun to read.  And thanks for putting so much work into this – it really was great over the course of ’08 to have House rankings from people whose opinions I value – reality-based opinions from people who understand the changing dynamics of how campaigns work, as opposed to inside-the-beltway types who make their predictions on how that candidate did while interviewing them.

  7. I can’t figure this one out.  The first thing to come to mind was Ready To Wear as in the fashion/designer sense.  Nope.  Right To Work as in the Republican phrase which means no Union Shops allowed.  Nope.  So what does the acronym stand for?

    PS it doesn’t show in a search on your site either.

  8. House polls this year. Nobody thought the Republicans were going to take it back, so there was no point in doing much public polling by district. We were a bit spoiled by Majority Watch and SUSA last cycle.  

  9. Oh you didn’t get 100% right, and I’m sure you’ll regret that forever (sarcasm).

    But I found the point to be the continuation of a conversation on politics, local politics, as to me the Senate and especially house acan reall get specific.  To that end, you succeeded tremendously.

    As a country we have been polarized for a while and probably still are, but the results indicate a swing in one direction.  One thing I learn from paying attention to a lot of this blog and the race rating in particular is the “all politics is local” mantra.

    Learning/reading about the specifics of a candidate or district is an enlightening process.  20 years ago it probably wasn’t even done by anyone except a person’s own district, now you can come here and read about almost every district.  

    We learn that party affiliation means different things in different states, and even at different levels (Running statewide as a Dem in Arkansas is easy, a Dem on the pres ticket in Arkansas is much different).  I learned that people actually care about the congressperson where I grew up (upstate NY, Gillibrand).  And so on and so on.

    I guess my one lone hope not yet fulfilled is that the partisanship decreases everywhere.  As a registered Dem like most posters on here (I assume), there is too much blaming in this world based on party or candidate.  And some of the arguments in blogs get so ridiculous there almost intolerable.  I think, though, its jsut representative of society.

    So, keep up the good work, and everyone on ehre has to remember we are all actually on the same team.  Every American is for a better America, a better democracy.  I’m certain of it.

Comments are closed.