The foundations of novelty

Yesterday, I was leafing through the journalist Charles Duhigg’s recent book Smarter, Faster, Better when my eye was caught by a discussion of a study conducted several years ago by two professors at Northwestern University. It dealt with the importance of unexpected combinations in creative thinking, which is a subject that is dear to my heart, and they emerged with some fascinating conclusions. Duhigg writes:

The researchers—Brian Uzzi and Ben Jones—decided to focus on an activity they were deeply familiar with: writing and publishing academic papers…They could estimate a paper’s originality by analyzing the sources authors had cited in their endnotes…Almost all of the creative papers had at least one thing in common: They were usually combinations of previously known ideas mixed together in new ways. In fact, on average, ninety percent of what was in the most “creative” manuscripts had already been published elsewhere—and had already been picked over by thousands of other scientists. However, in the creative papers, those conventional concepts were applied to questions in manners no one had considered before. “Our analysis of 17.9 million papers spanning all scientific fields suggests that science follows a nearly universal pattern,” Uzzi and Jones wrote. “The highest-impact science is primarily grounded in exceptionally conventional combinations of prior work yet simultaneously features an intrusion of unusual combinations.” It was this combination of ideas, rather than the ideas themselves, that typically made a paper so creative and important.

As Uzzi later explained in an interview with Duhigg: “A paper that combines work by Newton and Einstein is conventional. That combination has happened thousands of times. But a paper that combines Einstein and Wang Chong, the Chinese philosopher, that’s much more likely to be creative, because it’s such an unusual pairing.”

The late Arthur Koestler called this phenomenon “bisociation,” and its appearance here inspired me to look up the original paper, “How Atypical Combinations of Scientific Ideas Are Related to Impact.” Its most intriguing insight—which Duhigg mentions only in passing—is that not all combinations are equally useful. As you might expect, the study found that the majority of published scientific papers draw on a conventional set of sources, with most of their references occurring within a predictable subset of journals. Yet a novel combination of references in itself isn’t any guarantee of originality or importance. In fact, papers that have “high tail novelty” alone are actually less likely to be widely cited than papers that don’t stray far from conventional wisdom. The best combination, it seems, is a core of conventional work spiced up with a few unusual ingredients. An article on their research in the magazine of the Kellogg School of Management makes this point more explicit:

What’s interesting,” says Uzzi, “is most of the work done is conventional. And some of the work is truly novel. And the chances of either one of those classifications of papers being hits is about the same.” Only about five percent of research papers that draw from only very novel or only very conventional sources were among the most highly cited papers in the database. But there was a third category of research that had nearly twice the likelihood of making it big: papers that relied mostly on conventional combinations of sources but also included a small subset of highly novel ones. “It isn’t all about novelty or conventionality. It’s about both,” explains Jones, who was somewhat surprised by this result…“You want to be grounded in something that’s well understood and yet be adding in the piece that’s truly unusual. And if you do those two things [and] stretch yourself in both directions, then you radically increase your probability of hitting a home run.”

This point is technically present in Duhigg’s book, but it’s easy to miss, and it strikes me as the real takeaway here. Innovation doesn’t happen when you combine ideas haphazardly, but when you incorporate novel insights into a more conventional foundation. I’ve noticed this pattern in my own work. When I first started writing science fiction, my favorite method for generating ideas was to browse through a stack of science magazines and pick two or three articles at random, trusting that I would find a connection between them if I looked hard enough. My early novelette “The Last Resort,” for instance, was a combination of articles about lake eruptions, snowmaking, and the snake pits of Manitoba. “The Boneless One,” which was my first really good story, did the same with bioluminescence, octopus intelligence, and an expedition to catalog genetic material in the ocean. It was a reliable trick, and it served me well over the course of half a dozen stories. Around the time that I wrote “Stonebrood,” however, followed by “The Proving Ground” and my upcoming story “The Spires,” I began to get tired of that process, and I tried a different approach. These days, I usually start with one big subject or setting that I’d like to explore—wilderness firefighting, climate change in the Marshall Islands, bush piloting in Alaska. From there, I’ll look for an unusual angle that ties back into the main theme, often by searching the archives of science magazines until I come up with a promising hook. Instead of choosing a few random ideas and treating them equally, in other words, I start with a central premise that feels like it would make a good story and then look for unexpected offshoots. In some ways, this approach is riskier, since it that initial hunch is wrong, it’s easier to follow it into a dead end. But the results seem better. In the past, some of my stories, like “The Voices,” have had visible seams. The ones that I’m writing now are more of a piece, but they haven’t lost their ability to surprise me along the way, which is the main reason that I write them at all.

Obviously, this is a very minor example, and it probably isn’t all that interesting to anyone but me. But the notion that we should proceed by adding novelty to an established foundation, rather than by combining ideas purely at random, is a valuable one. It’s similar to the familiar principle that it’s hard to do interesting work across multiple disciplines until you’ve mastered one field well, both because of the habits of thinking that it teaches and the body of information that it provides. It was partly for this reason that Charles Darwin spent years studying barnacles, or cirripedes, as the scientist Thomas Henry Huxley observed:

The great danger which besets all men of large speculative faculty is the temptation to deal with the accepted statements of facts in natural science, as if they were not only correct, but exhaustive; as if they might be dealt with deductively, in the same way as propositions in Euclid may be dealt with. In reality, every such statement, however true it may be, is true only relatively to the means of observation and the point of view of those who have enunciated it. So far it may be depended upon. But whether it will bear every speculative conclusion that may be logically deduced from it is quite another question…The value of the Cirripede monograph lies not merely in the fact that it is a very admirable piece of work, and constituted a great addition to positive knowledge, but still more in the circumstance that it was a piece of critical self-discipline, the effect of which manifested itself in everything [Darwin] wrote afterwards, and saved him from endless errors of detail.

What the barnacles taught Darwin, in Huxley’s words, was “the speculative strain” that certain ideas would bear, and this applies as much to individual projects as to the work of a lifetime. Random combination can be a valuable tool, but it needs to be the right kind of randomness. Creativity, as Gregory Bateson wonderfully put it, often consists of “a raid on the random.” But like most raids, it’s more likely to succeed when it starts from a position of strength.

The secret of creativity

On Tuesday, in an article in The Daily Beast, I sampled some of the recent wave of books on consciousness and creativity, including Imagine by Jonah Lehrer and The Power of Habit by Charles Duhigg, and concluded that while such books might make us feel smarter, they aren’t likely to make us more creative or rational than we already were. As far as creativity is concerned, I note, there are no easy answers: even the greatest creative geniuses, like Bach, tend to have the same ratio of hits to misses as their forgotten contemporaries, which means that the best way to have a good idea is simply to have as many ideas, good or bad, as possible. And I close my essay with some genuinely useful advice from Dean Simonton, whom I’ve quoted on this blog before: “The best a creative genius can do is to be as prolific as possible in generating products in hope that at least some subset will survive the test of time.”

So does that mean that all other advice on creativity is worthless? I hope not, because otherwise, I’ve been wasting a lot of time on this blog. I’ve devoted countless posts to discussing creativity tools like intentional randomness and mind maps, talking about various methods of increasing serendipity, and arguing for the importance of thinking in odd moments, like washing the dishes or shaving. For my own part, I still have superstitious habits about creativity that I follow every day. I never write a chapter or essay without doing a mind map, for instance—I did the one below before writing the article in the Beast—and I still generate a random quote from Shakespeare whenever I’m stuck on a problem. And these tricks seem to work, at least for me: I always end up with something that would have occurred to me if I hadn’t taken the time.

Yet the crucial word is that last one. Because the more I think about it, the more convinced I am that every useful creativity tool really boils down to just one thing—increasing the amount of time, and the kinds of time, I spend thinking about a problem. When I do a mind map, for instance, I follow a fixed, almost ritualistic set of steps: I take out a pad of paper, write a keyword or two at the center in marker, and let my pen wander across the page. All these steps take time. Which means that making a mind map generates a blank space of forty minutes or so in which I’m just thinking about the problem at hand. And it’s become increasingly clear to me that it isn’t the mind map that matters; it’s the forty minutes. The mind map is just an excuse for me to sit at my desk and think. (This is one reason why I still make my mind maps by hand, rather than with a software program—it extends the length of the process.)

In the end, the only thing that can generate ideas is time spent thinking about them. (Even apparently random moments of insight are the result of long conscious preparation.) I’ve addressed this topic before in my post about Blinn’s Law, in which I speculate that every work of art—a novel, a movie, a work of nonfiction—requires a certain amount of time to be fully realized, no matter how far technology advances, and that much of what we do as artists consists of finding excuses to sit alone at our desks for the necessary year or so. Nearly every creativity tool amounts to a way of tricking my brain into spending time on a problem, either by giving it a pleasant and relatively undemanding task, like drawing a mind map, or seducing it with a novel image or idea that makes its train of thought momentarily more interesting. But the magic isn’t in the trick itself; it’s in the time that follows. And that’s the secret of creativity.

When reality isn’t good enough

Is reality a bore? Well, it depends on who you ask. Edward R. Tufte, in his wonderful book The Visual Display of Quantitative Information, devotes many pages to combating the assumption that data and statistics, being inherently dull, need to be dressed up with graphics and bright colors to catch the reader’s eye. The result, Tufte argues, is chartjunk, ink wasted on flashy design elements that have nothing to do with the information presented. Instead of investing resources in tarting up uninformative numbers, he says, one’s time is much better spent unearthing and analyzing relevant information. The best data, presented simply, will inspire surprise and curiosity, but only if the numbers are interesting and accurate, which requires its own kind of skill, ingenuity, and patience. Tufte sums up his case magnificently: “If the statistics are boring, then you’ve got the wrong numbers. Finding the right numbers requires as much specialized skill—statistical skill—and hard work as creating a beautiful design or covering a complex news story.”

Replace “statistics” with “stories,” and “numbers” with “facts,” and Tufte’s sound advice applies equally well to authors of nonfiction. It rings especially true in light of a number of recent controversies, both of which center on the question of when, if ever, reality should be manipulated for artistic reasons. One is the release of The Lifespan of a Fact, a book chronicling the five-year struggle between essayist John D’Agata and factchecker Jim Fingal over the accuracy of an essay finally published by The Believer. The other, of course, is the furor over a recent episode of This American Life, in which Mike Daisey’s account of his visit to a Chinese factory making components for Apple was revealed to have substantial fabrications. These are very different cases, of course, each with its own underlying motivations, but both are rooted in the assumption that reality, by itself, isn’t good enough. This led D’Agata and Daisey to embellish their stories with what might, at best, be termed “artistic” truth, but which can also be seen as the prose equivalent of chartjunk: falsehoods inserted to punch up the uncolorful facts.

D’Agata’s case is arguably the more instructive, because it’s founded on what appears to be a genuine artistic interest in blurring the lines between fiction and nonfiction. The original version of his essay, which uses the real suicide of a young man named Levi Presley as a means of exploring the culture of Las Vegas, contained countless departures from the facts, all purportedly for artistic reasons. Some were minor, such as changing the color of some vans from pink to purple because it scanned better, while others were fundamental: in his first paragraph, D’Agata refers to a series of strange events that occurred on the day of Presley’s suicide, including a tic-tac-toe contest against a chicken—none of which actually took place on the day in question. In other words, his list of unbelievable facts is literally unbelievable, because he made them up. In D’Agata’s hands, truth isn’t stranger than fiction; instead, fiction is exactly as strange as fiction, which raises the question of why we should care. In the end, his inability to find the real Las Vegas sufficiently colorful comes off as a failure of will, and the fact that he embellishes facts throughout the essay while keeping Levi Presley’s real name—presumably to gain a free artistic frisson from the circumstances of an actual suicide—seems like a particularly unfortunate case of wanting to have it both ways.

At least D’Agata has some kind of literary philosophy, however misguided, to justify his deviations from the truth (although it should be noted that most readers of The Believer presumably read his article as straight journalism). The same can’t be said of Mike Daisey, who altered the facts in The Agony and the Ecstasy of Steve Jobs to make it sound as if he personally witnessed events that occurred a thousand miles away, and to manufacture completely imaginary incidents for the sake of manipulating the audience. (In retrospect, it’s especially horrifying to hear Daisey’s voice grow soft and choked as he describes an injured factory worker’s first encounter with an iPad, a fictional incident that he describes as if it actually took place.) Daisey’s excuse, unlike D’Agata’s, is an emotional one: he wanted the audience to feel something, to be touched, implying that the true facts of his trip weren’t moving enough. Meanwhile, the legitimate journalism on Chinese factory conditions, as conducted by such reporters as Charles Duhigg and David Barboza of the New York Times, is far more fascinating, and it doesn’t depend on fabricated melodrama to make an impact.

As Tufte says, if the facts are boring, you’re using the wrong facts. But isn’t there a place for the judicious mingling of reality with fiction? Tomorrow, I’ll be talking more about this, and the importance of truth in labeling.