Quantification of Odds: a Tale of Miscommunication
Sherman Kent, PhD, CIA Cold War Senior Analyst

from Superforecasting, by Prof. Peter Tetlock and Dan Gardner

Tetlock, P. & Gardner, D. (2015). Superforecasting: The art and science of prediction. Random House, NY

In the late 1940s, the Communist government of Yugoslavia broke from the Soviet Union, raising fears the Soviets would invade. In March 1951 National Intelligence Estimate (NIE) 29-51 was published. “Although it is impossible to determine which course of action the Kremlin is likely to adopt,” the report concluded, “we believe that the extent of [Eastern European] military and propaganda preparations indicates that an attack on Yugoslavia in 1951 should be considered a serious possibility.” (emphases added) By most standards, that is clear, meaningful language. No one suggested otherwise when the estimate was published and read by top officials throughout the government. But a few days later, (Dr. Sherman) Kent was chatting with a senior State Department official who casually asked,“By the way, what did you people mean by the expression ‘serious possibility’? What kind of odds did you have in mind?” Kent said he was pessimistic. He felt the odds were about 65 to 35 in favor of an attack. The official was startled. He and his colleagues had taken“serious possibility” to mean much lower odds.

Disturbed, Kent went back to his team. They had all agreed to use “serious possibility”in the NIE so Kent asked each person, in turn, what he thought it meant. One analyst said it meant odds of about 80 to 20, or four times more likely than not that there would be an invasion. Another thought it meant odds of 20 to 80—exactly the opposite. Other answers were scattered between those extremes. Kent was floored. A phrase that looked informative was so vague as to be almost useless. Or perhaps it was worse than useless,as it had created dangerous misunderstandings. And what about all the other work they had done? Had they “been seeming to agree on five months’ worth of estimative judgments with no real agreement at all?” Kent wrote in a 1964 essay. “Were the NIE’s dotted with ‘serious possibilities’ and other expressions that meant very different things to both producers and readers? What were we really trying to say when we wrote a sentence such as this?”

- - - - - - - - - - - - - - - - - -

NB: One of my favorite aphoristic admonitions is: "If you can't give me a number, don't even talk to me!". ddavis