Sitting in a pub one night a dozen years ago, Charles Spence realized that he was in the presence of the ideal experimental model: the Pringles potato chip. Spence, a professor of experimental psychology at Oxford University, runs the Crossmodal Research Lab there, which studies how the brain integrates information from the five human senses to produce a coherent impression of reality. Very often, these modes of perception influence one another on the way to becoming conscious thought. For instance, scientists have long known that whether a strawberry tastes sweet or bland depends in no small part on the kinds of organic molecule detected by olfactory receptors in the nose. Spence had been wondering whether taste might be similarly shaped by sound: Would a potato chip taste different if the sound of its crunch was altered? To explore that question, he needed a chip with a reliably uniform crunch. The Pringle—that thin, homogeneous, stackable paraboloid—was perfect.
Over the next few weeks, Spence invited twenty research subjects to his basement lab and sat them in front of a microphone in a soundproof booth. There they were handed a pair of headphones and instructed to bite, one by one, into nearly two hundred Pringles original-flavor chips. After a single crunch, each subject spat out the chip and gave it a rating: crisper or less crisp, fresh or less fresh. The subjects could hear each crunch as it looped from the mike into the headphones. But, without letting the participants know, Spence funnelled the crunching noises through an amplifier and an equalizer, allowing him to boost or muffle particular frequencies or the over-all volume. About an hour later, released from the booth, each subject was asked whether he or she thought all the chips were the same.
The chips were identical, of course, but nearly all the volunteers reported that they were different—that some had come from cans that had been sitting open awhile and others were fresh. When Spence analyzed his results, he saw that the Pringles that made a louder, higher-pitched crunch were perceived to be a full fifteen per cent fresher than the softer-sounding chips. The experiment was the first to successfully demonstrate that food could be made to taste different through the addition or subtraction of sound alone. Spence published his results in the Journal of Sensory Studies, in 2004. The paper, written with a post-doc, Massimiliano Zampini, was titled “The Role of Auditory Cues in Modulating the Perceived Crispness and Staleness of Potato Chips.”
Within the small group of scientists interested in multisensory integration, the paper heralded a new direction for the field, a shift from teasing out the mechanics of audio-visual interaction to what Paul Breslin, an experimental psychologist at Rutgers University, described as “the new frontier” of oral perception. Outside the academy, the paper failed to generate any interest until 2008, when its authors were awarded the Ig Nobel Prize for Nutrition. The Ig Nobels are intended to “honor achievements that make people laugh, and then think,” but media coverage of Spence’s win focussed mostly on the former, with headlines that ranged from “BOFFIN GIVES EATERS SOUND ADVICE” to “WHY RESEARCH THAT?!” At first glance, the “sonic chip” experiment, as Spence fondly refers to it, does seem trivial. In reality, it was an elegant psychological trick, offering insight into the way the brain combines two separate sensory inputs—the crunching sound and the tactile oral sensation of a potato chip—into one multisensory perception. Spence lists the honor at the top of his curriculum vitae.
Before the sonic-chip breakthrough, Spence had worked almost exclusively on how an understanding of the neuroscience of audio and visual stimuli could help design better warning signals for drivers. (One of his insights—that sounds originating from behind a driver’s head will direct attention forward more quickly than sounds that come from the side—has found its way to market with the introduction of headrest-mounted speakers in 2015 Volvo FH trucks.) Afterward, Spence’s lab began studying the crunch of apples, the fizz of carbonated water, and the rustle of potato-chip bags. The vast majority of research into human sensory perception has been unimodal—focussed on understanding how each sense operates in isolation. Alongside sex, eating is one of the most multisensory of our activities; scientists have long claimed that much of what is perceived as flavor is actually filtered through the olfactory receptors, with taste buds playing a much smaller role. Spence goes further, arguing that in most cases at least half of our experience of food and drink is determined by the forgotten flavor senses of vision, sound, and touch.
Over the past decade, Spence has conducted a series of experiments that illustrate exactly that. Other researchers have joined him in exploring this new territory, but “Charles is a pioneer,” Francis McGlone, a neuroscientist at Liverpool John Moores University, told me. “His contribution to the field of cognitive psychology is seminal.” Breslin said, “He’s pushing the frontier in all kinds of ways that I wouldn’t have predicted.” In 1997, at the age of twenty-eight, Spence was invited to set up his own research lab at Oxford, and his Ig Nobel is just one in a long list of accolades, including a 2003 award from the European Society for Cognitive Psychology, in recognition of his “outstanding contribution to cognitive psychology in Europe.”
Along the way, Spence has found that a strawberry-flavored mousse tastes ten per cent sweeter when served from a white container rather than a black one; that coffee tastes nearly twice as intense but only two-thirds as sweet when it is drunk from a white mug rather than a clear glass one; that adding two and a half ounces to the weight of a plastic yogurt container makes the yogurt seem about twenty-five per cent more filling, and that bittersweet toffee tastes ten per cent more bitter if it is eaten while you’re listening to low-pitched music. This year alone, Spence has submitted papers showing that a cookie seems harder and crunchier when served from a surface that has been sandpapered to a rough finish, and that Colombian and British shoppers are twice as willing to choose a juice whose label features a concave, smile-like line rather than a convex, frown-like one.
It does not require an enormous leap of imagination to see how these kinds of cognitive insights could be incorporated into commercial packaging design, and, gradually, this is exactly what is happening. Americans derive a sizable proportion of their daily calories from food or drinks that are consumed directly from the package, and that is only expected to rise in tandem with the “snackification” of the Western diet. Marketing departments and product-design agencies have an extra incentive to enlist Spence’s findings in the cans, packets, tubs, and squeeze tubes that populate grocery-store shelves. We are accustomed to thinking of food and its packaging as distinct phenomena, but to a brain seeking flavor they seem to be one and the same.
“Dad always says, ‘Not the crisps again!’ ” Spence groaned, when I mentioned the Pringles experiment. We were walking back to his lab after lunch, during which he had criticized everything from the color of the rubber O-ring on the sparkling-water bottle to the sound of the Nespresso maker. Spence gives his family background the credit for his focus on finding practical applications for his research. “My parents never went to any school,” he said. “They always want to know, ‘What’s it for?’ ”
Spence’s mother and father both come from generations of showmen who travelled around northern Britain all year with the fair. “My great-grandfather was Randall Williams, King of Showmen,” Spence said, proudly. Williams is a legendary figure in British fairground history, known for his ghost show, a mirror-based illusion still used today in haunted houses, and for being among the first people to show moving pictures at a fair, in the eighteen-nineties. But Spence’s parents wanted a different life for their children, and although members of his extended family still work the fairgrounds of the North, he’s no longer in touch with them. By coincidence, one of the streets on our route was closed for the annual St. Giles’s Fair, a sensory riot of cotton candy, carrousel music, and flashing lights. “Too far south,” Spence said, before I could ask whether we might encounter members of the Spence clan manning the bumper cars or the coconut shy. “As soon as you become a flattie”—fairground slang for non-showmen—“that’s it, anyway.”
Spence favors research avenues that present themselves by happenstance and experiments that are built around mundane, domestic objects rather than lab rats. “When people ask what sort of research are you going to do next, I say, ‘I haven’t the faintest idea, because it depends who walks through the door,’ ” Spence said. Many of the entrants have been major food and beverage companies: Spence estimates that seventy-five per cent of his work is industry-funded. For a decade, he was a beneficiary of Unilever’s Cognitive Neuroscience Group, a short-lived corporate research department, led by Francis McGlone from 1995 to 2009. Since then, Spence has partnered with chefs such as Ferran Adrià and Heston Blumenthal, as well as with brands, museums, and culinary institutes—anyone who can offer him the chance to test a new hypothesis or attempt to replicate lab findings under more realistic conditions.
Back in the dingy chaos of his lab, Spence pointed out a boxful of unlabelled beverage cans, of all different sizes and shapes. A graduate student had spent the morning in the soundproof booth, recording the slightly different whoosh that each can made as she pulled the tab and popped it open. The cans had been supplied by Crown Holdings, an American company that produces one of every five beverage cans in the world; the recordings are to be used in a series of pre-trials to determine whether altering the particular pitch and tonal quality of a can’s opening hiss can make its contents seem fizzier or flatter, warmer or colder. If the student finds an effect, then the experiment will move to in-person testing, and then, if possible, to a large-scale proof under “ecologically valid” conditions, such as a bar or a restaurant.
“It’s looking at how auditory aspects of packaging can subliminally affect our perception of the product,” Cormac Neeson, the director of external affairs at Crown, told me. “And how then, we hope, we’d be able to maybe engineer those to give people an enhanced, improved experience through the packaging.” An all-new fizz-enhancing can is at least two years away, Neeson estimated, but he hopes that some of the study’s initial insights can be used sooner, to help his clients make more brand-appropriate choices—such as, say, a more forceful, “masculine”-sounding can for an energy drink. This draws directly on perhaps the most ubiquitous and tangible example of Spence’s research into sensory experience. In 2006, with funding from Unilever, Spence conducted a study to see whether altering the volume and pitch of the sound from an aerosol can would affect how a person perceives the pleasantness or forcefulness of a deodorant. Based on Spence’s findings, the company invested in a packaging redesign for Axe deodorant, complete with new nozzle technology. The underarm spray, which is targeted at young men, now sounds noticeably louder than the company’s gentler, female-targeted Dove brand.
“Basically, before it was pss and after it was ktsch-h-h-h-h,” Christophe Cauvy said. In 2013, Cauvy, then in charge of innovation for Europe at the advertising agency JWT (he now works independently), hired Spence as head of sensory marketing. “Charles’s research is immensely powerful and, I think, underused,” Cauvy told me. He quickly added, “I don’t want to sound too grand and say it’s the beginning of a revolution.”
Cauvy noted that the science of packaging has long sought to engage all the senses. As early as the nineteen-twenties, the product designer W. A. Dwiggins claimed that not only was the box more important than the thing inside it but our “digestive organs are trained to exude their juices, not at the sight of food, but at the sight of a rectangular pasteboard prism striped with alternating diagonal bands of white and red.” Later, Louis Cheskin, one of the great marketers of the twentieth century, coined the expression “sensation transference” to describe how making the hue of a 7UP can more yellow made the soda inside taste more lemony. By the nineteen-fifties, as Vance Packard recounted in “The Hidden Persuaders,” his exposé of the manipulative tricks used by marketers, the height of packaging sophistication was “a box that, when the entranced shopper picked it up and began fingering it, would give a soft sales talk, or stress the brand name. The talk is on a strip that starts broadcasting when a shopper’s finger rubs it.”
Of course, much of this marketing science was itself little more than sizzle. “Charles is different,” Cauvy told me. “Charles uses neuroscience to come up with what could be the ultimate sensory palette for packaging.” Still, marketers and packaging designers have arrived at many of Spence’s insights intuitively. “What Charles does,” Cauvy continued, “is minimize the risk of having the wrong intuition prevail.”
In his office, Spence maintains a rogues’ gallery of failed products—he had no part in their creation—and he gleefully demonstrated their sensory miscues. Sitting atop the filing cabinet was a special-edition white-colored can of Coke, introduced in 2011 to raise funds for endangered polar bears. It was withdrawn when consumers complained that Coca-Cola had also changed its secret formula. For Spence, the can is evidence of the power of a package’s color to alter the taste of the contents. His lab has repeatedly shown that red, the usual color of a Coke can, is associated with sweetness; in one experiment, participants perceived salty popcorn as tasting sweet when it was served in a red bowl. Next to the unlucky can sat an empty packet of Cadbury’s discontinued Koko-brand milk-chocolate truffles. As Spence has demonstrated, subjects associate a hard k sound with bitterness, while a softer b can make products seem sweeter. While the name “Koko” may not have single-handedly doomed the brand to failure, Spence argued that it might well have helped seal its fate.
In 2012, the snack company Mondelez, the owner of Cadbury’s, made another misstep. When it changed the classic rectangular chunks of Cadbury’s Dairy Milk into curved segments, customers complained that the chocolate tasted “too sugary” and “sickly.” Spence and other researchers have found that curved shapes can enhance sweetness. In one experiment, diners reported that a cheesecake tasted twenty per cent sweeter when it was eaten from a round white plate rather than a square one. In any case, Spence said, consumers are constantly, if unwittingly, proving his point that taste can be altered through color, shape, or sound alone. “These effects do exist,” he said. “The only question is whether and how we will use them.”
On a recent, unseasonably warm Friday, I met Spence at the Ace Hotel in Shoreditch, where a group of senior executives from Tesco, the biggest supermarket chain in the U.K., had gathered for a day of forecasting the future. Spence, who has the build of a former rugby player and a preference for coral-colored jeans, rocked back and forth on his toes as he introduced the group to the concept of multisensory design and its implications for retail. He described Axe’s aerosol rebrand, the mistaken Coke can, and flavor-boosting soundtracks he has worked on for Courvoisier and British Airways. He ended by focussing on the growing number of consumers older than seventy, whose senses of taste and smell have dulled. “They live in a very different taste world,” Spence said. But, he added, older people can still appreciate color, shapes, and sounds. “How can we optimize the sensory design of food for these silver palates?” he asked.
He noted that other researchers have shown that the elderly, when eating tomato soup, must add more than twice as much salt as a young person does in order to achieve the same taste. Why not mitigate that increased salt consumption, and its attendant health hazards, by presenting the soup in a blue container, a color that Spence has shown can make food seem significantly saltier? Similarly, experts estimate that sixty per cent of eighty-year-olds have an impaired sense of smell, sharply reducing their enjoyment—and thus, often, their intake—of food. In a live demonstration conducted in 2006 with the celebrity chef Heston Blumenthal, Spence found that when people were served a scoop of bacon-and-egg ice cream accompanied by the sound of sizzling bacon they described the taste of the ice cream as much more “bacony” than subjects whose consumption was accompanied by the clucking of chickens. This insight—that the appropriate soundtrack can intensify the flavor of a food—inspired Blumenthal’s iconic “Sound of the Sea” dish, for which diners at his restaurant, the Fat Duck, in Bray, are presented with an iPod loaded with a recording of crashing waves and screeching gulls to listen to while enjoying an artfully presented plate of seafood. The effect could be used similarly, Spence said, to design soundtracks that replace some of the lost flavor of food for the elderly.
Of course, a food company that uses visual or sonic cues to alter consumer experience could readily be accused of manipulation. In “The Hidden Persuaders,” Packard describes sinister marketing men trying to hypnotize the helpless housewife with carefully designed packages and cartons, as if “waving a flashlight in front of her eyes.” In “Packaged Pleasures: How Technology and Marketing Revolutionized Desire,” the historians Gary S. Cross and Robert N. Proctor point out that the evolution of food packaging is “at once a tale of sensorial enrichment and nutritional impoverishment,” in which products have been shaped to maximize sales and profits at the expense of health and well-being.
“It’s not about cheating,” Cauvy insisted. “This is really about a superior experience.” If Spence’s insights help to enhance the intensity of the eating experience, the argument goes, they will provide more enjoyable and more satisfying consumption, rather than simply increase it. Spence himself seems agnostic about the uses to which his research is put—he just wants it to be used.
Spence admits to being driven primarily by what he describes as the “drug-like thrill” of discovery. Nonetheless, he also believes that his research might change certain things, if not the world, for the better. He has met with the U.K. government’s Behavioural Insights Team to explain how his research might help in the fight against obesity by encouraging companies to use sensory seasoning to replace some of the salt and sugar in packaged foods. This year, he began working with a children’s cancer center in Spain, to experiment with plating, lighting, and acoustic tweaks that could counter the pervasive metallic taste and nausea that are common side effects of chemotherapy.
If there’s a criticism of Spence’s work, it’s that it is “atheoretical, trivial, and epiphenomenal,” Neil Martin, a psychologist at Regent’s University London, and a specialist in human olfaction and taste, told me. Spence proceeds by accretion rather than by explanation. He publishes at a rate that Francis McGlone described as pathological, co-authoring between fifty and a hundred papers a year, most of which describe a seemingly minor association without offering any causal mechanism. Recently, Spence, partly in response to his work’s perceived lack of depth, has started to work with philosophers on a series of experiments, workshops, and papers that aim at reconciling his research into crossmodal interaction with their theories of consciousness and reality. Eating, after all, is not just one of our most multisensory activities; it is also one of the most important activities for our continued survival. The obvious corollary of this, as the biologist J. Z. Young wrote, in 1968, is that “food is about the most important influence in determining the organization of the brain and the behavior that the brain organization dictates.”
“These things might seem trivial or esoteric,” Spence said. “But they do exist, and most of us share them, so they must be pretty fundamental.” Scientists—Spence included—do not yet fully understand exactly why these sensory interactions occur, or even where in the brain most take place. In a review paper published earlier this year, Spence listed the most common explanations that have been offered. Some correspondences may be universal. The connection between high-pitched sounds and sweet tastes is likely based on their shared oral configuration: an infant’s tongue moves outward and upward in response to sweet foods, as does an adult’s when he is singing falsetto. In other cases, the relationship might be culturally dependent, based on similar words being used across sensory modes—edges can look and feel sharp, but Cheddar can be described that way, too. The account that Spence frequently offers in his public lectures and media appearances is, at heart, a statistical one: cross-sensory correspondences can be attributed to the internalization of environmental patterns. Sweet, ripe fruit is often red; fresh foods usually make a louder crunch; and spiky salad leaves are frequently more bitter than their rounded counterparts.
Nonetheless, Spence’s own contributions as a scientist undoubtedly lie more in designing experiments than in analyzing the neuroscience behind his findings. One reason that the senses have almost always been studied in isolation is that it is remarkably difficult to disentangle them sufficiently to observe the effect that one might be having on another. Spence has managed to study them together by, as McGlone put it, “slipping in through the perceptual windows” created by sensory illusions.
His Pringles experiment is a telling example of this methodology. In designing it, Spence drew on an audio-tactile phenomenon known as the “parchment-skin illusion.” Described, in 1998, by the Finnish neuroscientist Veikko Jousmäki, the illusion can be experienced if you write on a chalkboard or rub sandpaper on wood, first with earplugs and then without. As the brain glues together the acoustic input and the tactile input from the hand holding the chalk or the sandpaper, it gives priority to the information from the ear. As a result, the same chalkboard or wood surface feels smoother when you wear earplugs to muffle the sound. By transposing illusions from one sense to another, Spence has begun to trace the hidden rules governing the interactions between them.
The evening after the Tesco meeting, Spence returned to London. He was due to give a talk at a beer festival behind King’s Cross station. In the previous seventy-two hours alone, he had spoken about his research with a branding agency, a group of Australian tourist-board representatives, a creative “salon,” two different museums, and a social-dining startup, not to mention the roomful of supermarket executives. Spence’s enthusiasm seemed undiminished, but he confessed that he was frustrated by the slow pace of corporate adoption. As he navigated his way across London, using a printout from Google Maps (he has no cell phone), he said that he had begun to fantasize about “chucking it all in” in order to set up an experimental restaurant—a space in which he could collaborate with chefs and designers to bring his vision of crossmodal dining to life, while conducting experiments on the public. A handful of lab members had started calling themselves the Crossmodalists, he said, and were in the process of drafting a manifesto.
Spence arrived five minutes before he was due onstage. In an unmarked tent opposite a long line of porta-potties, Spence plugged in his laptop and looked out over the empty folding chairs arrayed in front of him. Tipsy twentysomethings slowly drifted in, until, half an hour after the event was scheduled to start, Spence decided to begin. The microphone did not work, and the event organizers were nowhere to be found. Spence plowed on, making jokes and calling on his audience to guess the difference between Krug and cheap Cava from the sound that their respective bubbles made when bursting. Gradually, the tent filled up, and the drunken buzz decreased. Randall Williams, King of Showmen, would have been proud.
Having secured the audience’s attention, Spence asked people to sample a dark Welsh ale: one sip while listening to a light, tinkling xylophone composition, and the second to the sound of a deep, mellifluous organ. When the second piece of music stopped, the audience had fallen silent.
“Wow,” a girl near me in a vintage houndstooth dress said. I knew this particular trick of Spence’s—I had watched him perform it multiple times—but it still worked on me. With only a change in the background music, the deep-brown beer had gone from creamy and sweet to mouth-dryingly bitter.
As I watched the audience respond, it was hard not to conclude that Spence’s crossmodal correspondences were more than mere illusions. I was struck, too, by the contrast between the rest of the beer festival, which featured British youths in all their drunken glory—dancing on tables, fighting one another, throwing up—and the attention and thoughtful discussion that accompanied Spence’s presentation. For all his psychological insights, his real contribution may lie in allowing people to appreciate anew the rich, multisensory experience of eating.
Spence, meanwhile, was moving on to his next event, a Skype presentation that he was to give to a conference in New Zealand in a couple of hours. Unnoticed on the stage, he unplugged his laptop, finished his beer, and headed off to catch the last train to Oxford. ♦
