Crossing the Rhine

Two groups of very smart people are looking at the exact same data and coming to wildly different conclusions. Science hates that.

—Katie M. Palmer, Wired

In the early thirties, the parapsychologist J.B. Rhine conducted a series of experiments at Duke University to investigate the existence of extrasensory perception. His most famous test involved a deck of Zener cards, variously printed with the images of a star, a square, three waves, a circle, or a cross, in which subjects were invited to guess the symbol on a card drawn at random. The participants in the study, most of whom were college students, included the young John W. Campbell, who displayed no particular psychic ability. At least two, however, Adam Linzmayer and Hubert Pearce, were believed by Rhine to have consistently named the correct cards at a higher rate than chance alone would predict. Rhine wrote up his findings in a book titled Extrasensory Perception, which was published in 1934, and I’m not going to try to evaluate its merits here. What I will note is that attempts to replicate his work were made almost at once, and they failed to reproduce his results. Within two years, W.S. Cox of Princeton University had conducted a similar run of experiments, of which he concluded: “There is no evidence of extrasensory perception either in the ‘average man’ or of the group investigated or in any particular individual of that group. The discrepancy between these results and those obtained by Rhine is due either to uncontrollable factors in experimental procedure or to the difference in the subjects.” By 1938, four other studies had taken place, to similar effect. Rhine’s results were variously attributed to methodological flaws, statistical misinterpretation, sensory leakage, or outright cheating, and in consequence, fairly or not, parapsychological research was all but banished from academic settings.

Decades later, another study was conducted, and its initial reception was very different. Its subject was ego depletion, or the notion that willpower draws on a finite reservoir of internal resources that can be reduced with overuse. In its most famous demonstration, the psychologists Roy Baumeister and Dianne Tice of Case Western University baked chocolate chip cookies, set them on a plate next to a bowl of radishes, and brought a series of participants into the room. All were told to wait there, but some were allowed to eat the cookies, while the others were instructed to snack only on the radishes. Then they were all given the same puzzle to complete—although they weren’t told that it was impossible to solve. According to the study, students who had been asked to stick to the radishes spent an average of just eight minutes on the puzzle, while those who had been allowed to eat the cookies spent nineteen minutes. The researchers concluded that our willpower is a limited quantity, and it can even be exhausted, like a muscle. Their work was enormously influential, and dozens of subsequent studies seemed to confirm it. In 2010, however, an analysis of published papers on the subject was unable to find any ego depletion effect, and last year, it got even worse: an attempt to replicate the core findings, led by the psychologist Martin Hagger, found zero evidence to support its existence. And this is just the most notable instance of what has been called a replication crisis in the sciences, particularly psychology, with one ambitious attempt to duplicate the results of psychological studies, the Reproducibility Project, finding that only about a third could be reproduced.

But let’s consider the timelines involved. With Rhine, it took only two years before an attempt was made to duplicate his work, and two more years for the consensus in the field to turn against it decisively. In the case of ego depletion, twelve years passed before any questions were raised, and close to two decades before the first comprehensive effort to replicate it. And you don’t need to be a psychologist to understand why. Rhine’s results cut so radically against what was known about the brain—and the physical universe—that accepting them would have required a drastic overhaul of multiple disciplines. Not surprisingly, they inspired immediate skepticism, and they were subjected to intense scrutiny right away. Ego depletion, by contrast, was an elegant theory that seemed to confirm ordinary common sense. It came across as an experimental verification of something that we all know instinctively, and it was widely accepted almost at once. Many successful studies also followed in its wake, in large part because experiments that seemed to confirm it were more likely to be submitted for publication, while those that failed to produce interesting results simply disappeared. (When it came to Rhine, a negative result wouldn’t be discarded, but embraced as a sign that the system was working as intended.) Left to itself, the lag time between a study and any serious attempt to reproduce it seems to be much longer when the answer is intuitively acceptable. As the Reproducibility Project has shown, however, when we dispassionately pull studies from psychological journals and try to replicate them without regard to their inherent interest or plausibility, the results are often no better than they were with Rhine. It can leave psychologists sounding a lot like parapsychologists suffering through a crisis of faith. As the psychologist Michael Inzlicht wrote: “Have I been chasing puffs of smoke for all these years?”

I’m not saying that Rhine’s work didn’t deserve to be scrutinized closely, because it did. And I’m also not trying to argue that social psychology is a kind of pseudoscience. But I think it’s worth considering whether psychology and parapsychology might have more in common than we’d like to believe. This isn’t meant to be a knock against either one, but an attempt to nudge them a little closer together. As Alex Holcombe of the University of Sydney put it: “The more optimistic interpretation of failures to replicate is that many of the results are true, but human behavior is so variable that the original researchers had to get lucky to find the result.” Even Martin Hagger says much the same thing: “I think ego-depletion effect is probably real, but current methods and measures are problematic and make it difficult to find.” The italics, as usual, are mine. Replace “human behavior” and “ego depletion” with “extrasensory perception,” and you end up with a concise version of the most widely cited justification for the resistance of such abilities to scientific verification, which is that these phenomena are real, but difficult to reproduce. You could call this wishful thinking, and in most cases, it probably is. But it also raises the question of whether it’s possible to have a meaningful phenomenon that can’t be reproduced in a laboratory setting. Regardless of where you come down on the issue, the answer shouldn’t be obvious. Intuition, for instance, is often described as a real phenomenon that can’t be quantified or replicated, and whether or not you believe this, it’s worth taking seriously. A kind of collective intuition—or a hunch—is exactly what determines what results the scientific community is likely to accept. And the fact that this intuition is so often wrong means that we need to come to terms with it, even if it isn’t in a lab.