What should you do when people you respect offer conflicting business or personal advice? First, realize that nearly all real-world advice contains a mixture. There are real facts, presumed facts, personal values and cognitive inferences, all of them intermingled, often in complex ways, perhaps invisible to the advisers themselves. Second, try to untangle this mess using a divide-and-conquer approach.

Here are some examples to guide you:

The Denver Bullett Battle

A couple of decades ago a highly-charged controversy confronted Denver's mayor: should he allow the police to use deadlier, hollow-point bullets. The officers were being outgunned by criminals and felt vulnerable. But critics argued that the police should not become judge, jury and executioner all at once. Hollow point bullets are meant to enhance "stopping power" --the ability to immobilize suspects-- but cause far more bodily harm. Various citizens groups disliked hollow point bullets and street protests escalated, with each side marshaling its own ballistics experts.

To solve the brewing crisis, the mayor reached out to Professor Kenneth Hammond, a decision expert from the University of Colorado. Hammond interviewed each side and found that the opposing ballistics experts did not disagree much on the injury potential, stopping power or ricochet risk of the bullets. Their divergence stemmed from different implicit tradeoffs they made between safety, suspect rights, bullet costs and the cops' desire not to be outgunned.

Likewise, citizens demonstrating in the streets argued heatedly about ballistic kinetics, but without much expertise. The professor then build a simple decision model that clearly separated facts--based on the ballistics experts' judgments-- from societal values, which a policy committee would rule on as part of standard democratic procedures already in place. This led to a weighted ranking of all bullets considered and solved the crisis to the satisfaction of most due to transparency and rigor. Also, there was unanticipated solution: the bullet that came out on top was different from the old ones used and the hollow point ones desired by the police. The highest scoring bullet improved stopping power for the cops without increasing bodily harm.

Attacking Osama bin Laden’s Compound

But what to do you do if the experts don't agree on the facts? President Obama faced this problem in the White House's legendary Situation Room when confronted by conflicting advice over mounting a commando raid to capture terrorist leader Osama Bin Laden. As recounted in Mark Bowden's book The Finish: The Killing of Osama bin Laden , the president listened intently as Central Intelligence Agency officers discussed the identity of a very tall man in a mysterious Pakistani compound in the city of Abbottabad. The CIA's team leader tells the president he is nearly certain the subject is bin Laden. "He put his confidence level at 95%," Bowden writes. Another CIA officer agreed. Others, however, were less sanguine. "Four senior officers at the Directorate of National Intelligence place their confidence level at about 80%; some were as low as 40 or even 30%." Yet another officer was 60% confident that bin Laden was in the compound.

"OK, this is a probability thing," the president sighed in response, according to one account. From a decisionmaking viewpoint, that's 'so far, so good,' since Obama had decomposed the problem into facts versus values. He relied on experts for the best available judgments about the factual matter of tall man's identity without burdening them with bigger decisions about whether to attack the compound. That critical, historic policy decision would clearly be his.

But after listening further to the wide-ranging probability estimates about the identity of the mystery man in the compound, Obama remarks with some frustration: "Look guys, this is a flip of the coin. I can't base this decision on the notion that we have don't have any greater certainty than that." That summary silenced everyone--and reveals how inadequately even the smartest leaders often handle uncertainty and probabilities. A simple wisdom-of-the-crowd average in that room would put the odds around 70%, even though Obama described it as a 50-50 situation. Was he revising down the group's probability estimate because he felt they might perhaps be overconfident? Was he just conservative himself in this case? Or was Obama just using a figure of speech to signal that much was unknown, akin to flipping a coin? And does it really matter if the true probability is 50% or 70% in deciding whether to launch an American attack in Abbottabad, Pakistan?

The Takeaway for Leaders

There are some key leadership lessons to be learned from the bin Laden case about interpreting ambiguous advice. First, Obama views the disagreement among his trusted experts as problematic and frustrating; after all, they are supposed to be 'best informed,' so how can they differ so much? Second, instead of delving even more deeply into the reasons underlying their divergent views, Obama focuses on the average odds provided, rather than the considerable uncertainty surrounding them--where the real learning lies. Third, he does not quite know how to weigh the relative importance of each expert's view. Obama may privately hold some advisers in higher regard than others regarding their substantive expertise in this matter, but does he really know how well each person can express his or her opinion probabilistically?

Unfortunately, we don't really know the final determinants of why Obama ultimately decided to attack the compound. We do know that it was a nearly complete success. After debate and exchange of views has occurred, the question of how to combine expert opinions usually remains since consensus is rare. Synthesizing different views into a single, subjective probability judgment about a key event is not simple, as explained in the book Superforecasting, which discusses this case and others. What is paramount, however, is not just how best to aggregate diverse opinions but to first delve deeply into the root causes of divergent opinions. Try to decompose them into differences about facts vs. values, then discuss each source of conflict, and lastly, recombine the pieces into a better judgment overall.

The latter can be done by averaging final opinions, favoring some advisors over others, using statistical updating procedures, or forming an intuitive overall judgment. This is challenging perhaps but far from impossible. We'll discuss the conditions favoring one approach over another in our next post. Hint: much depends on the issues involved, the organizational context, the ultimate aims and the personal preferences of those responsible for the decision.


Co-authored with Philip Tetlock, professor at the University of Pennsylvania in psychology, political science and business (Wharton) plus co-author of Superforecasting, Crown, 2015.