Swing State Project

« MD-Sen: Fuck You, Michael Steele | Main | VA-Sen: Webb's Website is Up »

Friday, February 10, 2006

What's a Prediction Worth?

Posted by DavidNYC

Not much, apparently.

A commenter somewhere on the site (sorry, can't find it now) recently remarked about the fact that political prognosticators were way off the mark when it came to predicting the 1994 elections, the year of the GOP landslide. It got me wondering, "How bad were they?" So I managed to dig up an old Hotline item which tabulated the election eve predictions of 21 different so-called "experts". I can't link to it, but I've reproduced the relevant parts.

First, a little background. Prior to the 1994 elections, the Democrats held a comfortable 56-44 margin in the Senate and a huge 258-176 margin in the House. (Bernie Sanders had been kickin' it as the lone independent since 1991.) Those numbers are almost hard to believe today. After the election, the Dems were reeling. The GOP held a 52-48 advantage in the Senate (which would soon grow to 54-46 as two Democrats defected) and a 230-204 lead in the House (Sanders managed to barely hang on).

But no one saw it coming. Here, take a look:

Pundit	Senate Margin	House Margin
Alan Abramowitz	GOP +2	DEM +28
Charlie Cook	GOP +4	DEM +8
Ronald Lester	DEM +2	DEM +26
Frank Luntz	GOP +4	DEM +6
Mary Matalin	GOP +6	GOP +4
Chris Matthews	GOP +6	DEM +10
Ralph Reed	GOP +2	DEM +20
Bill Schneider	Even	DEM +18
Marty Tolchin	Even	DEM +6
Ron Walters	GOP +2	DEM +18
Fred Wertheimer	Even	DEM +14
Mark Shields	GOP +2	DEM +12
Al Hunt	GOP +2	DEM +10
Bob Novak	GOP +6	GOP +6
Margaret Carlson	GOP +2	DEM +22
Doug Bailey	GOP +6	DEM +10
Fred Barnes	GOP +6	DEM +6
Morton Kondracke	GOP +4	GOP +14
John McLaughlin	GOP +4	GOP +2
Jack Germond	Even	DEM +26
Eleanor Clift	DEM +2	DEM +22
Average Prediction:	GOP +2.57	DEM +11.24
Actual Outcome:	GOP +4	GOP +25

Most pundits predicted that the GOP would take the Senate - fully fifteen were within ±2 of the right answer (which was GOP +4). But oh man, no one even came close to getting the House right. Only four people even said the GOP would emerge victorious, and none of them predicted the eventual 25-seat margin. As far as the House results were concerned, these predictions were almost entirely worthless.

Now, you might respond by saying that fewer seats are in play today, and that predictions have greater utility now because the House picture is less volatile. In turn, I'd point you to this New Yorker review of a fascinating new book called "Expert Political Judgment," by Berkeley psychology prof Phil Tetlock. Tetlock did a systematic study of expert predictions over a period of twenty years (talk about patience). His findings are pretty extraordinary:

Tetlock also found that specialists are not significantly more reliable than non-specialists in guessing what is going to happen in the region they study. Knowing a little might make someone a more reliable forecaster, but Tetlock found that knowing a lot can actually make a person less reliable.

“We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly,” he reports. “In this age of academic hyperspecialization, there is no reason for supposing that contributors to top journals—distinguished political scientists, area study specialists, economists, and so on—are any better than journalists or attentive readers of the New York Times in ‘reading’ emerging situations.”

And the more famous the forecaster the more overblown the forecasts. “Experts in demand,” Tetlock says, “were more overconfident than their colleagues who eked out existences far from the limelight.”

Bottom line: Beware predictions, especially from high-profile experts. That's not to say we shouldn't try to make any - hell, that's half the purpose of the political blogosphere. It just means we all have to maintain a strong skepticism - and keep that salt handy.

Posted at 08:59 PM in Elections | Technorati

Trackback Pings

TrackBack URL for this entry:
http://www.swingstateproject.com/mt/mt-track-ssp.cgi/2149

Listed below are links to weblogs that reference What's a Prediction Worth?:

» Experts Got It Wrong in 1994 from Taegan Goddard's Political Wire
Swing State Project looks back at pundit predictions before the 1994 midterm elections and notes almost no one saw the Democrats losing both the House and the Senate. "Bottom line: Beware predictions, especially from high-profile experts. That's not to... [Read More]

Tracked on February 13, 2006 03:33 PM

Comments

Oops. Actual Senate results were GOP +8, David. (44 --> 52>

Posted by: PeteyP at February 11, 2006 01:42 PM | Permalink | Edit Comment | Delete Comment

PeteyP, the number is the margin, not the number of seats gained. After the election the Senate was 52 Republicans and 48 Democrats, so the margin was GOP +4.

But it does look like four people, not three, predicted the Republicans would take the House: Matalin, Novak, Kondracke, and McLaughlin.

Posted by: KCinDC at February 11, 2006 02:55 PM | Permalink | Edit Comment | Delete Comment

KC, right on both counts. Thanks for catching that.

Posted by: DavidNYC at February 11, 2006 06:18 PM | Permalink | Edit Comment | Delete Comment

I completely agree. Charlie Cook and Larry Sabato's updated predictions for this fall are boilerplate, business-as-usual evaluations that will only come to pass in the event that voters head to the polls in November 2006 with identical partisan allegiances as they did in 2002 and 2004. Sabato "leans Republican" even in races like IN-08 (Hostettler vs. Ellsworth) and CT-02 (Simmons vs. Courtney) where incumbents look to be highly vulnerable. In a way, I can understand why they choose to play it so safe even when shifting political winds point to a Congressional shakeup (1994 and again in 2006). They don't want to seem like cheerleaders for one side or the other and lose credibility. But just as right-wingers like Novak and Kondracke were probably best-positioned to sense the coming tsunami in favor of the GOP in 1994, I'll be most interested in the predictions of Mark Shields and Eleanor Clift this time around than predictors straitjacketed to either "objectivity" or helping to minimize losses for their team.

Of course, the last four election cycles have ended with one party closing the deal far better than was expected the day before. The Democrats vastly outperformed expectations in 1998 and 2000, while the Republicans sailed above the bar in 2002 and 2004. There's always the risk of thoroughly embarrassing ourselves by highballing our own performance. Sooner or later, we may learn our lesson and walk away from the increasingly fruitless predictions game.....but I doubt it.

Posted by: Mark at February 11, 2006 07:15 PM | Permalink | Edit Comment | Delete Comment

Hm does anyone know what Clintions approvals where during the 1994 election?

Posted by: D in FL. at February 12, 2006 02:23 PM | Permalink | Edit Comment | Delete Comment

The night before the 1994 elections, no one predicted the House would fall to the GOP and this was even in the face of last day polling showing a strong shift in sentiment to the GOP. most commentators thought the GOP would hold a 52-48 edge in the Senate and no one thought voter anger would have a tsunami effect. Remember this was the year that Tom Foley, the then Speaker, lost his seat as did Jack Brooks of Texas who had held on to his seat for almost 30 years and lost it to a novice, Steve Stockman. The Clinton approvals were probably then as low as Shrub's are now.

Another factor to remember is that the early 90's were years of very strong voter disatisfaction, a feeling that lasted for a number of years. Dick Thornburgh, Poppy Bush's Attorney General, lost a Pennsylvania Senate race to a Democratic college president, Harris Wofford in 1991. In 1992, Ross Perot got, I think, almost 19% of hte vote. Whether that level of voter anger exists today, I am not qualified to address but if there is to be the kind of Dem gains that the GOP experienced in 1994, it's going to have to run awfully deep.

Posted by: charlton at February 13, 2006 09:51 AM | Permalink | Edit Comment | Delete Comment

KC: Thanks for pointing that out - I was too quick for my own good!

Posted by: PeteyP at February 13, 2006 05:08 PM | Permalink | Edit Comment | Delete Comment