The Problem with Pundits and the Steve Jobs Effect

In the last few days leading up to the recent presidential election, long time political observers such as Peggy Noonan, Steve Forbes and Dick Morris were predicting that Mitt Romney would win in a landslide.

They were wrong, of course.  Horribly, drastically, overwhelmingly wrong about everything from the shape of the electorate, voter turnout and, of course, the direction of the result in the face of massive, publicly available evidence that showed Barack Obama as the clear favorite.

InTrade markets gave 7 to 3 odds that the President would prevail.  Nate Silver’s FiveThirtyEight blog (which analyzes polling data) calculated a 91% chance of an Obama win.  How could they get it so wrong?  It turns out that the answer lies not so much in partisan politics, but in the inherent flaws of our basic human decision making machinery.

Tetlock’s Revenge

It’s tempting to see the dismal failure of pundits in factional terms.  Given their sometimes hysterical backlash against any information that contradicted their convictions, right wing talking heads might seem completely irrational.  In retrospect, their misplaced confidence in their cause seems positively crazy.

However, according to Philip Tetlock, who embarked on a 20 year study of political pundits, expert predictions of all ideological persuasions are no better than flipping a coin. In other words, for all the time they spend studying an issue (and in fact, to some extentbecause they spend so much time on it), they might as well just throw darts.

Yet, there is something more going on here.  These pundits in particular not only got it wrong, but chose to ignore the most salient evidence – a puzzling, as well as troubling development.  Why would they reject the most obvious answer in favor of obscure arguments about poll skews and media bias?

The Tail of Two Systems

We can find part of the solution to the pundit puzzle in Nobel laureate Daniel Kahneman’s excellent book Thinking, Fast and Slow.  He describes our decision making process as a competition between two competing systems:

System 1: This is our more instinctual, automatic system.  It relies on rules of thumb, called heuristics, which enable us to act quickly and decisively.  It is fast and frugal, using little time and energy to arrive at a conclusion.

Incidentally, system 1 is very active in politics.  Studies by political scientists have found that voters often make judgments of candidates based on “rapid, unreflective inferences” about their appearance.

System 2: This system reflects our more rational side.  We use it when we stop, think, make difficult calculations and weigh facts, which takes considerably more time and effort than simply going on instinct.  We only engage our second system when we realize that the first one is falling short.

Kahneman argues that we have a tendency to substitute tough system 2 questions (such as how to analyze an enormous amount of polling data) for an easier system 1 question (i.e. how one feels about the possibility of an outcome).  That, as we shall see, is where the trouble starts.

Substituting One Question For Another

We ‘re more comfortable with judgements we make on the basis of system 1 because it feels more natural.  We don’t live our lives based on polls or artificial markets, but mostly by gut feelings.  Does this seem like a good neighborhood?  Can I trust this salesman? Does that item on the menu look tasty?

These are the questions we ask ourselves everyday and no amount of controlled tests or sophisticated mathematical models will do us any good.  We go with what feels right, picking up subtle cues that will sway us one way or another because of how they jibe with our internalized experience.

Take a look at what longtime Wall Street Journal political columnist Peggy Noonan had to say about why she felt so strongly about a Romney win:

All the vibrations are right. A person who is helping him who is not a longtime Romneyite told me, yesterday: “I joined because I was anti Obama—I’m a patriot, I’ll join up But now I am pro-Romney.” Why? “I’ve spent time with him and I care about him and admire him. He’s a genuinely good man.” Looking at the crowds on TV, hearing them chant “Three more days” and “Two more days”—it feels like a lot of Republicans have gone from anti-Obama to pro-Romney.

Yep.  There’s her system 1 kicking in.  She can feel it in her bones, therefore it must be right, because what you feel is not only very real, it is reality itself.  Once your system 1 becomes convinced, it becomes very easy to reject the relatively weak impulses of system 2, which she does just a few paragraphs later.

Is it possible this whole thing is playing out before our eyes and we’re not really noticing because we’re too busy looking at data on paper instead of what’s in front of us? Maybe that’s the real distortion of the polls this year: They left us discounting the world around us.

Of course, professionals can not judge based on feelings alone, they need facts to back them up, but that’s a relatively minor problem, for we can always find data to support a position.  After all, no president had been re-elected with such a high unemployment rate and 2008 was a wave election, reflecting a misshapen electorate.  Right?

When these comforting morsels agree with our system 1, they become very convincing. Behavioral economists call this confirmation bias and we all fall prey to it.

The Confidence Game

When the “vibrations are right” as they were for Peggy Noonan, doubt begins to fade and a dangerous confidence sets in.  Conservative columnist Ross Douthat describes what happens next:

You could see this belief at work in the confidence with which many conservatives insisted that the Obama presidency was not only embattled but self-evidently disastrous, in the way so many voices on the right sought to raise the ideological stakes at every opportunity, in the widespread conviction that the starker conservatives made the choice between left and right, the more votes they would win.

You could also see this conviction shaping the punditry and predictions that issued from conservatives in the days leading up this election. It was remarkable how many analysts not normally known for their boosterism (I’m thinking of Michael Barone and George Will in particular) were willing to predict that Romney would not only win but win sweepingly, capturing states that haven’t gone Republican since Reagan. But even less starry-eyed conservatives — like, well, myself — were willing to embrace models of the electorate that overstated the Republican base of support and downplayed the Democrats’ mounting demographic advantage.

Pundits tend to be homophilic.  They form schools of thought and think tanks, which creates an echo chamber for friendly conclusions.  There is nothing more satisfying than spending time with people who passionately agree with us.  Our brains release oxytocin, giving us the ultimate feeling of well being and reinforcing our system 1 even further.

We no longer merely think that we’re right, we’re absolutely sure of it.  Pundits who convey this confidence are especially convincing, which is what makes them compelling to audiences and increases the public’s demand for their services.  Surety sells.

The Steve Jobs Effect

We live in a data driven, technological age and this offends our system 1.  Data is never completely clear and we require complicated mathematics to digest it.  It is no accident that the polling analyst Nate Silver called his new book The Signal and the Noise.

We’ve found a hero in Steve Jobs, who famously never hired consultants or used research surveys.  He was more of an artist than an engineer and conjured big ideas.  He operated by instinct and that seems admirable.  Highly regarded columnists Michael Gerson and David Brooks both remarked that the reliance on polling data diminished big ideas.

This is the most dangerous trick that system 1 plays on us, for in many highly technical fields, system 1 is an enormous asset, allowing us to act quickly and confidently.  We don’t, after all, want a surgeon consulting a database when blood begins to spurt or a fireman running models in excel while flames engulf our homes.

However, as I explained in a previous post, these problems are not equivalent.  Just like Steve Jobs spent many hard years studying design, surgeons and fireman personally confront thousands of life or death situations.  That kind of immediate, direct feedback is wholly absent in the political realm.  Elections are rare and we experience them by proxy.

The Visceral Abstract

The issue goes far beyond political prognostication.  As I’ve argued before, we increasingly live in a world of the visceral abstract, where positively wacky ideas that are completely foreign to our experience have taken a central role in our everyday lives.  However, we don’t directly confront them, which makes it hard to take them into account.

Einstein’s theory of relativity says that time and space are illusions, which only have relevance to our particular perspective.  Quantum mechanics tells us that while we are most likely here, we could be over there; or on the dark side of the moon for that matter.

Preposterous?  Maybe, but our GPS navigation systems and our iPhones are living proof that these things are true.

It is only through our relatively boring and tedious system 2 that we can understand this abstract world.  The “vibrations” that stirred Peggy Noonan’s soul gave her no insight into the feelings of millions of freethinking individuals across hundreds of miles, just as the comfortable look and feel of our iPhone tells us nothing about subatomic physics.

And that’s the problem with pundits.  They are paid to dazzle us with their surety and it is that same confident conviction that makes them so very likely to get it wrong.

image credit: notionscapital.com

Don’t miss a post (4,900+) – Subscribe to our RSS feed and join our Innovation Excellence group!

Greg Satell a consultant who concentrates on media, marketing and innovation. Check out at his site, Digital Tonto and follow him on twitter @digitaltonto