The myth of experience

During World War II, the legendary physicist Freeman Dyson—who wasn’t even twenty years old at the time—served as a scientist with the Operational Research Section of the Royal Air Force Bomber Command, which attempted to use statistical methods to analyze and improve airborne operations. In his memoir Disturbing the Universe, Dyson tells the following story:

I was engaged in a statistical study to find out whether there was any correlation between the experience of a crew and their chance of being shot down. The belief of the Command, incessantly drummed into the crews during their training and impressed on the public by the official propaganda machine, was that a crew’s chance of surviving a mission increased with experience. Once you get through the first five or ten missions, the crews were told, you will know the ropes and you will learn to spot the German night fighters sooner and you will stand a much better chance of coming home alive. To believe this was undoubtedly good for the boys’ morale. Squadron commanders, all of them survivors of many missions, sincerely believed that they owed their survival to their personal qualities of skill and determination rather than to pure chance. They were probably right. It had been true in the early years of the war that experienced crews survived better.

Dyson notes that a study conducted before his arrival had confirmed the official doctrine of the importance of experience, and that its conclusions had been “warmly accepted” by everyone involved. When he took another look at the problem, however, the results weren’t as encouraging:

Unfortunately, when I repeated the study with better statistics and more recent data, I found that things had changed…My conclusion was unambiguous: the decrease in loss rate with experience which existed in 1942 had ceased to exist in 1944. There were still many individual cases of experienced crews by heroic efforts bringing home bombers so badly damaged that a novice crew in the same situation would almost certainly have been lost. Such cases did not alter the fact that the total effect of all the skill and dedication of the experienced flight crews was statistically undetectable. Experienced and inexperienced crews were mown down as impartially as the boys who walked into the German machine gun nests at the battle of the Somme in 1916.

So what happened? According to Dyson, the theory within the Operational Research Section, which later turned out to be correct, was that the sudden irrelevance of experience was due to the introduction of a new technological factor: the Schräge Musik, or upward-firing cannons, which Germany had recently installed on many of its fighter planes. Allied bomber crews had been trained to scan the sky for enemy fighters—but the area below the aircraft was a blind spot. And once the Germans had vertical guns, which could be aimed upward while the fighter approached quietly from below, the experience of the air crews became useless.

Before I finish this story, it’s worth reflecting on what it tells us so far. We’re naturally inclined to attribute successful outcomes to talent or experience, rather than luck, because it flatters us into thinking that we’re more in control of our lives than we really are. This is true of artists, hedge fund mangers, and—as the events of this year have emphatically demonstrated—career politicians, and it means that we need to be particularly skeptical of anecdotal evidence that makes experience seem more important than it actually is. There’s also the unavoidable fact that even if experience is genuinely useful, a simple technological change can make nonsense of it. The introduction of firearms into medieval Japan threatened to render an entire system of values meaningless overnight: a child with a gun could kill a samurai from a distance. (Instead of discarding their culture, the Japanese made what, in some ways, was a more sensible choice: they decided to proceed as if guns didn’t exist for as long as they possibly could.) Finally, there’s the implication that when this sort of crisis occurs, you can’t solve it by advancing along the same lines of progress as before. Instead, you have to throw out whole sections of the rulebook, which is what happened in Dyson’s case. The Operational Research Section proposed that the number of crew members in each fighter be reduced from seven to five, and that the two gun turrets be removed completely. This would lighten the plane and allow it to fly fifty miles an hour faster, an advantage that would more than make up for the loss of its gunners: in night combat, speed was a far more decisive factor than firepower.

Of course, there was no guarantee that any of this would work, but Dyson observes that it had one undeniable advantage: “Even if the change did not result in saving a single bomber, it would at least save the lives of the gunners.” When the group brought the proposal to their section chief, who would decide whether or not to pass it up the pipeline, this was the result:

Our chief took a dim view of our suggestion that bombers might survive better without gun turrets. This was not the kind of suggestion that the commander in chief liked to hear, and therefore our chief did not like it either. To push the idea of ripping out gun turrets, against the official mythology of the gallant gunner defending his crewmates, and against the massive bureaucratic inertia of the Command, would have involved our chief in a major political battle. Perhaps it was a battle he could not have hoped to win. In any case, the instinct of a career civil servant told him to avoid such battles. The gun turrets remained in the bombers, and the gunners continued to die uselessly until the end of the war.

As for the chief, according to Dyson, his prudent unwillingness to deliver bad news to his commander “earned him the expected promotion at the end of the war and led to the inevitable knighthood.” In other words, he successfully navigated a system of incentives that was designed to reward experience whether it deserved it or not—a bureaucracy. And from his point of view, he was right.