Tuesday, November 29, 2005

previous entry | main | next entry | TrackBack (0)

Who needs experts?

Louis Menand has a glowing review in the New Yorker of Philip Tetlock's latest opus, Expert Political Judgment: How Good Is It? How Can We Know?. Some highlights:

It is the somewhat gratifying lesson of Philip Tetlock’s new book... that people who make prediction their business—people who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtables—are no better than the rest of us. When they’re wrong, they’re rarely held accountable, and they rarely admit it, either. They insist that they were just off on timing, or blindsided by an improbable event, or almost right, or wrong for the right reasons. They have the same repertoire of self-justifications that everyone has, and are no more inclined than anyone else to revise their beliefs about the way the world works, or ought to work, just because they made a mistake. No one is paying you for your gratuitous opinions about other people, but the experts are being paid, and Tetlock claims that the better known and more frequently quoted they are, the less reliable their guesses about the future are likely to be. The accuracy of an expert’s predictions actually has an inverse relationship to his or her self-confidence, renown, and, beyond a certain point, depth of knowledge. People who follow current events by reading the papers and newsmagazines regularly can guess what is likely to happen about as accurately as the specialists whom the papers quote. Our system of expertise is completely inside out: it rewards bad judgments over good ones....

Tetlock also found that specialists are not significantly more reliable than non-specialists in guessing what is going to happen in the region they study. Knowing a little might make someone a more reliable forecaster, but Tetlock found that knowing a lot can actually make a person less reliable. “We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly,” he reports. “In this age of academic hyperspecialization, there is no reason for supposing that contributors to top journals—distinguished political scientists, area study specialists, economists, and so on—are any better than journalists or attentive readers of the New York Times in ‘reading’ emerging situations.” And the more famous the forecaster the more overblown the forecasts. “Experts in demand,” Tetlock says, “were more overconfident than their colleagues who eked out existences far from the limelight.”...

The experts’ trouble in Tetlock’s study is exactly the trouble that all human beings have: we fall in love with our hunches, and we really, really hate to be wrong....

The expert-prediction game is not much different. When television pundits make predictions, the more ingenious their forecasts the greater their cachet. An arresting new prediction means that the expert has discovered a set of interlocking causes that no one else has spotted, and that could lead to an outcome that the conventional wisdom is ignoring. On shows like “The McLaughlin Group,” these experts never lose their reputations, or their jobs, because long shots are their business. More serious commentators differ from the pundits only in the degree of showmanship. These serious experts—the think tankers and area-studies professors—are not entirely out to entertain, but they are a little out to entertain, and both their status as experts and their appeal as performers require them to predict futures that are not obvious to the viewer. The producer of the show does not want you and me to sit there listening to an expert and thinking, I could have said that. The expert also suffers from knowing too much: the more facts an expert has, the more information is available to be enlisted in support of his or her pet theories, and the more chains of causation he or she can find beguiling. This helps explain why specialists fail to outguess non-specialists. The odds tend to be with the obvious.

There are intriguing implications for understanding world politics that deserves a post of their own, but suffice it to say that Tetlock's findings will probably warm the cockles of every political blogger out there.

posted by Dan on 11.29.05 at 10:41 PM


Who expects political analysists to be 'right' as often as financial analysts are, whose stock predictions are notoriuosly bad? After all, much more relevant information is available about publicly traded companies, and the analysts predictions are about very discrete events like stock price movements.

For an exploration of this theme from a interesting prespective - that maybe the very few people who DO get these prediction right over long periods of time are merely lucky and not wise, see Malcom Gladwell's New Yorker article "Blowing Up" here - http://www.gladwell.com/2002/2002_04_29_a_blowingup.htm

posted by: Jos Bleau on 11.29.05 at 10:41 PM [permalink]

So if Tetlock is right, then he is as full-of-it as any expert, because by writing this book he is saying he is an expert expert, which means he's wrong- so if he's wrong then he's dead-on like all the experts, but then wait a second, I'm confused.

Oh wait I get it. It's common knowledge that experts are full of it, so Tetlock is just agreeing with the commonfolk, which means he can be correct. Which means his book won't tell you anything you didn't already suspect.

I hope his book has some serious figures to back up his claims or is filled with wit and delightful, well-crafted anecdotes. Otherwise it sounds hypocritical. And I'm no expert, so you can take that opinion to the bank, smartie pants.

posted by: Rob C on 11.29.05 at 10:41 PM [permalink]

I remember thinking along these lines pre-Iraq:

Tetlock also has an unscientific point to make, which is that “we as a society would be better off if participants in policy debates stated their beliefs in testable forms”

I also thought I was way naive to think that way. A war on Al Qaeda or Osama was testable ex ante. A war on "terror" is intrinsically not testable.

On the other hand, suppose you become attached to just one metric above all others, clinging to it even as circumstances change, going on talk shows and it is the only metric that matters ...

posted by: Robert Bell on 11.29.05 at 10:41 PM [permalink]

> "will probably warm the cockles of every political blogger out there."

Of course it will. Confirmation bias is one of the reasons people get things wrong so often. If a study agrees with my hunch, I swallow it uncritically. If it disagrees, I'll nitpick it to pieces.

How do I get a talking head job? I read the NYTimes and I've got lots of free time.

posted by: Dude on 11.29.05 at 10:41 PM [permalink]

Just what I need, warm cockles!

posted by: David Pinto on 11.29.05 at 10:41 PM [permalink]

I actually believe in experts. I apply this term to anyone who knows a great deal more than I do about a given subject.

Now, if you ask any expert about a subject different than (sometimes even slightly different than) his or her area of expertise, the value of the response you get may be no better than you could have gotten by directing your inquiry to someone picked at random from the phone book. It is also true that an expert who may have a great deal to say at some times may not at others. No talk show guest ever answered a question with the words "I have nothing useful to say about this subject right now" -- because guests are flattered to be asked, and want to be asked again. They will say almost anything rather than have to say nothing.

Finally, an expert may have excellent judgment as well as extensive knowledge about a given subject. This is not a requirement, however, and it sometimes takes multiple exposures to any particular expert to ascertain whether he or she is really a reliable source for analysis, or merely for data. Where television is concerned, once an expert is established as sufficiently entertaining his judgment and even the extent of his knowledge may not exclude him from being trotted out to give his learned opinion on any old thing.

posted by: Zathras on 11.29.05 at 10:41 PM [permalink]

what a blowhard.

that is the dumbest thing ive ever heard and is just another strike at the continually growing disregard people have - wrongfull so - for political science. econ is suffering this to a degree, but not as much as poli sci.

sorry, this guy is just plain stupid. im not going to address this further.

posted by: Derek on 11.29.05 at 10:41 PM [permalink]

The point of political experts on TV is not to be right. It is to articulate different versions of the conventional wisdom. Yes, if you are a smart reader of the new york times, you can develop your own, better point of view. But if you are not a smart reader of the NYT, only an average cable viewer, you can choose one of the two sides on a cable talk show, and adopt it as your own, without having to develop your own more subtle take.

In other words, their function is to show us what opinions to adopt as fashionable or appropriate to our political perspectives. It is not to make us think deeply.

posted by: DK on 11.29.05 at 10:41 PM [permalink]

Not sure if the same is true with political pundits but I have been called on occasion to make comment about market matters. I can say this about the process by which an expert gets quoted. You will only be quoted if what you say fits the story the journalist wants to tell or thinks they should tell. Usually they will question more sources than they need and cherry pick the bits and pieces they want/need out of a pool of quotes. Hence as an expert you quickly learn what makes the cut. What makes the cut generally might have a bit of novelty but generally not too far from consensus. If you say something to far off you will be cut.

posted by: asiequana on 11.29.05 at 10:41 PM [permalink]

enlarger patch penis

posted by: enlarger patch penis on 11.29.05 at 10:41 PM [permalink]

prosolution pill

posted by: prosolution pill on 11.29.05 at 10:41 PM [permalink]

When there is no alternative but to look back at the entries made here, I wonder why I keep trying.

posted by: penis pills on 11.29.05 at 10:41 PM [permalink]

dsnpklc ztfq tedhxmsi edsqtjxk lmwqb zecn ncbvw

posted by: kctdjveqz kgcz on 11.29.05 at 10:41 PM [permalink]

Post a Comment:


Email Address:



Remember your info?