Separate the Bet From the Result

The Core Insight

Twenty-six seconds remained in Super Bowl XLIX. Seattle trailed by four points with the ball on second down at the New England one-yard line. Pete Carroll called a pass, Russell Wilson threw it, and New England intercepted.

Four outlets called it the worst or dumbest play call in Super Bowl history the next morning. Benjamin Morris and Brian Burke ran the numbers instead. Of sixty-six passes attempted from an opponent's one-yard line that season, zero had been intercepted. Over the previous fifteen seasons the interception rate in that spot was about 2 percent. Resulting is equating the quality of a decision with the quality of its outcome.

Carroll drew the distinction himself on television four days later, calling it the worst result of a call ever. Annie Duke retired from poker in 2012 with a World Series bracelet and more than 4,000,000 dollars in tournament winnings. Most people grade a decision by how it turned out. Duke runs an exercise with executives, and every one of them names best and worst results rather than decisions. She argues that outcomes carry too much luck to grade anything, and that the process is the only gradeable part.

Resulting survives because people think they are playing chess. Chess has no hidden information and almost no luck. A loss means a better move existed and you did not see it, so outcomes track decision quality. Poker is a game of incomplete information played under uncertainty over time. You can make the best available decision at every point and still lose.

The Framework

The quality of our lives is the sum of decision quality and luck.

A bet is a decision about an uncertain future. The dictionary definition Duke uses carries five words: choice, probability, risk, decision, and belief. Ordering the chicken instead of the steak is a bet, and declining to bet is a bet. In most decisions the opponent is not another person. You are betting against every future version of yourself you did not choose.

The book runs each bet through a fixed pipeline, and every stage is a place where it goes wrong.

• Beliefs arrive by hearing, and vetting happens later or never.
• Every bet is priced by the beliefs sitting underneath it.
• Outcomes get fielded into a skill bucket or a luck bucket, and that call is another bet.
• A truthseeking group does the fielding you cannot do alone.
• Mental time travel pulls the future and the past into the decision in front of you.

Key Ideas

You Believe First and Vet Later

The usual account of belief runs in three steps: you hear something, you test it, then you believe it or reject it. Daniel Gilbert summarized centuries of research in a 1991 paper with the order reversed. People find believing easy and doubting hard, and belief works more like involuntary comprehension than like assessment.

Two years later his subjects read statements about a criminal defendant, color coded true or false. Under time pressure or a small distraction they made more errors, and the errors ran one way. They treated every statement as true, whatever its label.

Correction lands weakly. Hollyn Johnson and Colleen Seifert had subjects read messages about a warehouse fire, including a closet of paint cans and pressurized gas. A correction five messages later said the closet was empty, and subjects still blamed burning paint for the fumes.

Every bet rests on beliefs, so the cost compounds. Americans cut a quarter of their calories from fat in one generation and replaced it with carbohydrates. The advice drew partly on research the sugar industry paid for. David Ludwig wrote in JAMA that obesity tripled, type 2 diabetes rose many-fold, and the long decline in cardiovascular disease flattened.

Being Smart Makes the Blind Spot Bigger

Albert Hastorf and Hadley Cantril showed film of a rough 1951 Princeton and Dartmouth game to students at both schools, who counted the infractions. Princeton students saw Dartmouth commit twice as many flagrant penalties as Princeton. Dartmouth students saw an equal number from each team. Their 1954 paper concluded that people behave according to what they bring to the occasion.

Motivated reasoning drives the split. A lodged belief makes you notice confirming evidence, leave it unchallenged, and work to discredit whatever contradicts it.

Intelligence makes the loop stronger. West, Meserve, and Stanovich tested subjects for seven cognitive biases in 2012, and cognitive ability did not reduce the blind spot. In six of the seven, the more sophisticated participants showed larger blind spots. Awareness of your own biases did not help either.

Dan Kahan ran the sharper version. Subjects analyzed made-up data about an experimental skin treatment, and performance tracked numeracy. The researchers kept the data identical and relabeled it as concealed-weapons bans and crime. Political belief drove the reading, and the more numerate subjects made more mistakes than less numerate subjects who shared their beliefs. Skill with numbers is skill at bending numbers toward what you already believe.

Confidence Belongs in Percentages

Duke replaces the word confident with a number. Rate a belief from zero to ten, where zero means certain it is false and ten means certain it is true. A six means 60 percent, so 40 percent of the time the belief turns out wrong. Where a point estimate resists, give a range: Elvis died somewhere between forty and forty-seven. Better information tightens the range.

The mechanic pays twice. Updating stops being humiliating, because moving from 58 percent to 46 percent is a smaller admission than moving from right to wrong. Stated uncertainty makes you more credible and invites people to refine the belief with you.

Duke once called one hand a 76 percent favorite and the other 24 percent, and the 24 percent hand won. A spectator told her she was wrong, which is what 24 percent looks like when it arrives. A forecast between zero and 100 percent cannot be proved wrong by one future failing to arrive. Nate Silver gave Trump between 30 and 40 percent in the week before the 2016 election.

Outcomes Land in the Wrong Bucket

Every outcome gets fielded like a ball in the outfield. If making the same decision again predictably produces the same outcome, the outcome came from skill. If it came from things outside your control, like other people, the weather, or your genes, it came from luck.

Self-serving bias fields it for you. Robert MacCoun studied accounts of car accidents. In 75 percent of the accounts, victims blamed someone else for their injuries. In multiple-vehicle accidents, 91 percent of drivers blamed someone else. In single-vehicle accidents, 37 percent of drivers still found someone else to blame.

Phil Hellmuth holds fourteen World Series bracelets and told ESPN that luck is the only thing stopping him from winning every tournament. Duke's point is that he said it out loud.

Your bad outcomes are bad luck and other people's are their fault. Cubs fans blamed Steve Bartman for deflecting a foul ball in the 2003 playoffs. The Cubs still led 3-0 with five outs to go, and the Marlins then scored eight runs. Seven of them followed an error by the Cubs shortstop, who escaped the blame for a decade.

A Truthseeking Group Runs on a Charter

David Letterman asked Lauren Conrad on air in October 2008 whether she was the problem. The read was sharp and the forum was wrong, because she never agreed to that exchange. Truthseeking works when the other person chooses it.

Three people are enough: two to disagree and one to referee. The charter has three rules.

• A focus on accuracy over confirmation, with truthseeking and open-mindedness rewarded inside the group.
• Accountability, which every member agrees to in advance.
• Openness to a diversity of ideas.

Philip Tetlock and Jennifer Lerner found the condition that produces open-minded thought. People reason well when they learn in advance that they answer to an informed audience whose views are unknown.

Erik Seidel enforces it at the table. He declines to hear a hand story whose point is bad luck, and offers unlimited time on strategy. Howard Lederer let Duke ask him only about hands she had won, and she had to name a point where she made a mistake. Talking about a win hurts less, so the new habit trains faster.

Accountability needs teeth. Duke set a loss limit of 600 dollars at her stakes, and the limit held because her group asked about it. Anna Dreber ran a betting market on the Reproducibility Project replication attempts, using expert traders and forty-four studies. Traditional peer review by those experts was right 58 percent of the time, and the same experts betting money were right 71 percent.

Prospective Hindsight Pays Thirty Percent

Gary Klein summarized a 1989 experiment by Deborah Mitchell, J. Edward Russo, and Nancy Pennington. Imagining that an event already happened raises the ability to correctly identify reasons for future outcomes by 30 percent.

Backcasting starts from a headline saying the goal was achieved, and the group works backward through the decisions and breaks that got them there. A premortem starts from the headline saying the goal was missed. A company planning to double market share from 5 to 10 percent in three years writes both headlines. The failure frame lets people name a problem without playing the naysayer, and the branches add to 100 percent.

Positive fantasy on its own does damage. Gabriele Oettingen calls the alternative mental contrasting. Women in a weight-loss program who held strong positive fantasies about slimming down lost twenty-four pounds less than the women who pictured obstacles.

Scouting the futures finds the branch nobody saw in Carroll's call. A pass gives Seattle three plays to score instead of two. The price is an interception probability of 2 to 3 percent, and a run carries a fumble probability of 1 to 2 percent. Very few commentators found that advantage even with days to analyze it. The 2 to 3 percent branch reads as 100 percent once it becomes the past, which is hindsight bias.

Practical Applications

Name your best and worst decisions of the last year, then check whether you named decisions or results. A thoughtful process that ended badly is a bad result. Changing your behavior on the strength of it is how a person concludes they drive better drunk.

Put a number on your next three claims before you argue them. Say the number to the person who disagrees, and write down the moment you move from 58 percent to 46 percent.

Ask for advice with the outcome removed. Give the details of the decision and withhold both the result and your own conclusion. Outcome-blind analysis spread through parts of particle physics and cosmology because knowing the answer contaminates the analysis.

Build the group of three before you need it, and get the charter agreed in advance. Name the exceptions too, like a night reserved for complaining about bad luck. An experienced player plays about 20 percent of hands and studies the other 80 for free.

Write both headlines before the next plan is approved, then rank the options by expected value. The After-School All-Stars ranked grant applications by award size until the arithmetic reordered the stack. A 50,000 dollar grant that lands 70 percent of the time is worth more than a 100,000 dollar grant that lands 25 percent.

Who This Is For

Founders deciding under hidden information with slow feedback get the most from this book. Hiring, pricing, and market entry all resolve late and noisily, and each one gets graded on the result.

Skip it if your work has short clean feedback loops and little hidden information. The book is thin on computing probabilities and long on why you must state them.

Two soft spots sit under the evidence. Duke argues from a poker career, where stakes are defined and a hand ends in two minutes. The transfer to a market with none of those properties is asserted more than it is shown. Much of the cited psychology comes from areas with replication trouble. Her own sharpest example is a betting market built on replication attempts. The truthseeking group also fails where founders need it most. Advance agreement to accuracy rarely survives a reporting line, because the people who owe you dissent are the people you pay.

The Decision

Pick a decision you already made and whose result you already know. Write down what you knew at the time, the options in front of you, and the probability you gave each one before it resolved.

Then check which thing you graded, the process you ran or the branch that grew. A gap between the two makes the fix mechanical. State the next decision as a bet, attach a number, and put it in front of someone who agreed in advance to argue.

Life is one long game, and a single hand grades nothing.