The Dark Side of Optimism

This text was originally published in Dagens Perspektiv.

"Think positive," they say, but is a lack of positive thinking really our biggest problem? No, the late Daniel Kahneman would probably have said.

When the Guardian asked him which cognitive bias he would eliminate if he could wave a magic wand, he answered "overconfidence" – the tendency we humans have to place too much trust in our own abilities and to be overly optimistic about the projects we undertake.

For even though many of us regularly experience low self-confidence and suffer from impostor syndrome, the general tendency among most people is that we are a little too optimistic.

Not According to Plan

A good example comes from my work at Stiftelsen Dam (the Dam Foundation). The foundation invests close to 400 million Norwegian kroner annually, partly in Norwegian health research. In their applications for funding, applicants must state how long the research project will take. On average, those who receive funding say it will take 33 months to complete the project.

How long does it actually take? 76 months, well over twice as long as planned!

And this is just the time spent. In addition, major changes often occur in other parts of the project – fewer participants are recruited than planned, more withdraw from the study than expected, equipment costs more, and project staff have children or resign.

When the project is completed, the project leader must fill out a final report where we ask "To what extent did the project go according to plan?" On average, the project leaders answer 75 percent.

In other words, as a foundation we receive 75 percent of what we were promised, and it takes more than twice as long to get it. We can live with that, because we know this applies to everyone who funds research. Things never go as planned, and we take a calculated risk when making such investments.

Learning Little from Experience

The strange thing is that we all have the experience of getting it wrong, yet we don't learn from it.

One study showed that parties who decline a settlement offer and choose to go to trial receive less than what they turned down in 85 percent of cases. This is in itself a striking example of overconfidence, but the study was also interesting because it examined whether the lawyers' level of experience had any impact on the success rate. You would think that a lawyer who has repeatedly made misjudgments would become better at preventing such errors. However, the study showed that experience made no difference.

Nor do ordinary people seem to learn. A large study showed that age had no impact on the degree of overconfidence. It is not the case that inexperienced youth are hit harder by overconfidence than those who have many more decades of lived experience.

How Confident Are You?

How many people live in the beer and football city of Munich?

Instead of a single number, I want your answer in the form of an interval that you are 90 percent confident contains the correct answer. If it were Oslo, I would for example answer 580,000 to 720,000. I am about 90 percent sure that the actual number of inhabitants in Oslo falls within that range, but I'm not entirely certain. I would have been guaranteed to be right if I answered "0 to 1 billion," but that's not the task.

So what is your answer?

I have created a quiz consisting of 10 such questions and have received just over 10,000 responses. This gives me insight into an entirely different and much more interesting question: How often are people right when they say they are 90 percent sure?

On average, it turns out that people are only right 36 percent of the time!

This never ceases to fascinate me. "I am 90 percent sure I'm right" turns out in this context to actually mean "I'm right 36 percent of the time."

In case you're wondering, approximately 1.5 million people live in Munich.

Only Bad?

Is it only negative to have too much confidence in your own judgments and success?

A little too much optimism has clear positive sides. First of all, it feels good and is associated with better mental health.

Secondly, a slightly exaggerated belief in success also leads people to launch projects that will be completed, but that would never have been started if the cost had been known in advance. Some such projects can turn out to be good investments. I have frequently heard people with doctoral degrees say "If I had known how demanding it was, I would never have done it, but I'm glad I did it anyway."

Elon Musk is widely known for launching seemingly insane projects and grossly missing the mark on many of his assumptions (self-driving Teslas were supposed to be here as early as 2016), but his optimism and drive have obviously also been absolutely crucial in making PayPal, Tesla, and SpaceX happen at all.

The answer depends on the degree of overconfidence. Having an excessively inflated belief in your own success will logically lead to a graveyard of overambitious projects.

What Can You Do?

Although there are positive sides to overconfidence, there is little doubt that it should be reined in when the stakes are high. It would have been nice, for example, if Norway's Follobane railway tunnel hadn't cost more than three times the original budget.

Fortunately, it is possible to do something about overconfidence. The method is called "reference class forecasting," or "the outside view."

Let's say you want to open a pizza restaurant. You think it will go well because you make great pizza and work hard. But is that a good basis for starting? Who opens a restaurant without believing they make good food and are willing to work hard?

Using "the outside view," you should not start from your own perspective and your own assessment, but instead look at your idea the way someone on the "outside" would. In short, the method is about starting your assessment by looking at what has historically happened in similar situations, and it works in two steps.

Step 1: Start with a Historical Reference

Begin by looking for similar decision situations to the one you are in. If you're considering opening a pizza restaurant, you could for example look for public statistics on how new businesses in the hospitality industry perform.

At Statistics Norway (SSB) you would see that only 40% of businesses in this industry still exist five years after they were founded.

The reference you have found should be your starting point. In the case of the pizza restaurant, your starting point should be that there is a 40 percent chance it will still exist after five years.

Step 2: Adjust the Estimate Based on Judgment and Local Knowledge

Once the starting point is established, it's time to adjust the estimate. There may be several reasons to believe that your pizza restaurant could do better than the average Norwegian hospitality business. Perhaps you have an especially good pizza sauce recipe and there happens to be a pizza wave sweeping the country?

Remember, however, that there are usually also good arguments for why the estimate should be even more conservative. The fact that a pizza wave is sweeping the country probably also means you'll face many competitors. Perhaps it's also particularly difficult to find workers in your area.

Make a list of all such arguments and adjust the estimate accordingly. Don't look at everything as a whole, but try to attach a numerical value to each argument. When everything is listed and evaluated as in the table, you are left with a final estimate that should form the basis of the decision you need to make. Are you willing to bet on opening a pizza restaurant that has a 51 percent chance of surviving after five years?

Argument	Adjustment (percentage points)	Survives after five years
Starting point	—	40%
Especially good pizza sauce	+ 8	48%
Pizza wave	+ 10	58%
High competition	- 4	54%
Labor shortage	- 3	51%

Closing Thoughts

Your judgments are and will remain vulnerable to various types of cognitive errors. Overconfidence is one of those that hits us hardest and which, despite its positive sides, carries significant costs in the form of budget and time overruns. But it doesn't have to be this way, and "the outside view" is one way to limit the damage.