Menus Subscribe Search
(ILLUSTRATION: MÁGOZ)

(ILLUSTRATION: MÁGOZ)

Replicate This

• February 26, 2013 • 4:00 AM

(ILLUSTRATION: MÁGOZ)

Do classic psychological studies published in high-profile journals hold up? The Reproducibility Project aims to find out.

There are few psychological effects better known—or more widely accepted—in academic halls than what is called semantic priming. Show a person a simple stimulus, something as unremarkable as a photograph of a cat. Let some time pass, then ask that same person to list as many words as possible that start with the letter c. This person is more likely not only to come up with the word cat, but to mention catlike animals such as cougars and cheetahs, because he was initially primed with that one little kitty cat.

Priming’s reach, of course, stretches far beyond cognitive tests. Therapists use it to help treat patients with depression during therapy sessions. Advertisers count on commercials to prime us to buy key brands during our trips to the mall or the grocery store. Priming is considered an underlying mechanism in stereotyping. And the word has become part of our cultural lexicon, too. We talk about how we are “primed” to feel, to want, to need, to talk. Priming is everywhere.

And yet, many of the classic studies that led us to our current understanding of priming have never been replicated. In fact, the few attempts to reproduce the results that we have taken at face value for so long have failed. In late 2012, that led Daniel Kahneman, noted Princeton University psychologist and author of the best-selling book Thinking Fast and Slow, to write an open e-mail to the entire priming-research community. He wrote, “Your field is now the poster child for doubts about the integrity of psychological research. Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published.” Kahneman’s solution? A new research protocol whereby cooperating labs attempt to check and replicate each other’s studies. This is the only way, he argues, to separate the scientific wheat from the chaff.

But accuracy and integrity issues are not limited to studies about semantic priming. They plague the whole psychological community. Research replication, an essential feature of good science—the element that allows truth to shine through the experimental brume—has simply not been a priority in today’s “publish or perish” climate. And we’re now learning that many well-publicized studies can’t be replicated. (This may be due to, say, incorrect or inappropriate analysis of results, or a sample size that is too small.) When studies can be reproduced, there is little incentive for scientists to do so.

“Journals are geared toward publishing new stuff, and that new stuff tends to be overwhelmingly positive results,” says Eric Eich, editor of the journal Psychological Science. “Discovery work, across the board, tends to be valued higher than doing confirmation or replication work.”

So with reputations hanging in the balance, what can be done?

That’s the question Brian Nosek, a psychologist at the University of Virginia, has been pondering. Last year, he launched the crowdsourced Reproducibility Project, with the express mission of replicating psychological studies published in high-profile journals. “Academic science is open, transparent—people are supposed to be able to see the evidence and basis for different claims and then evaluate them,” Nosek tells me. “Science values truth above all else.”

The first step: to find enough psychologists willing to forgo the prestige of discovery work. Nosek reached out to his colleagues across the country—psychologists who, like him, had been decrying the lack of replication in the field at conferences and meetings over the past decade. He challenged each of them to try to reproduce a single study from a sample of those published in three eminent psychological journals. By spreading the work around, and mitigating the difficulties and costs involved with replication, Nosek argued, the field could finally get an idea of what social psychology’s reproducibility rate really is.

The project has been met with overwhelming praise—publicly, at least. Nosek’s original recruitment e-mail went viral, reaching a larger audience than he ever imagined. Now more than 100 scientists across more than 40 global institutions have joined the reproducibility mission, using their own laboratory resources, and the project’s guidelines, in attempts to replicate studies. About 20 replication attempts have already been completed.

Georg Jahn, a psychologist at the University of Greifswald, in Germany, recently replicated a 2008 study that looked at the importance of attention when learning the associations between adjacent and nonadjacent items—a skill that has implications for the way people learn language and grammar. Jahn says the replication was very straightforward, and the results confirmed the original study’s findings.

But reproducing studies, and determining whether original results hold up, is not always so clear-cut. Privately, scientists have voiced concerns about what counts as a true replication. What if the exact same materials or methods are not used? What if the study is run in a setting with slightly more—or less—controls? The devil, as they say, is in the details.

Michael Frank, a developmental psychologist at Stanford University, ran across these issues as he and his students worked on the replications of several survey-type studies. One was a paper titled “Why People Are Reluctant to Tempt Fate.” (pdf) Lead author Jane Risen, at the University of Chicago, and her coauthors documented that individuals, reacting to a series of “what if?” scenarios, responded that it was bad luck to tempt fate—even when they didn’t believe in fate.

Risen’s goal was to find out how and why people can believe things they know are false. Frank’s preliminary results suggest that the magical thinking described in the original paper was not evident in the new study. Risen, though, believes Frank’s students’ work was not a real replication. The Stanford group ran the study on the general population instead of on a subset of students—and did so using the Internet, whereas the original study was conducted in person. “Because the study involved participants imagining themselves being called on in a college class, it was important that it be run with student participants who could relate to the story,” Risen said.

Nosek is quick to tell me that a study could fail to replicate an original result for many reasons: study methods may be just different enough, or a study’s sample not quite large enough to be statistically valid. He admits that he and his project partners are learning as they go—and before making any declarations about a researcher’s success or failure to reproduce a study, they are evaluating every project on a case-by-case basis.

Accuracy is key. If they get a low reproducibility rate, the outcomes, obviously, can have weighty consequences. Donors may be less inclined to fund studies in the field, Nosek explains, or people may lose their trust in psychology altogether.

Kayt Sukel
Kayt Sukel is the author of Dirty Minds: How Our Brains Influence Love, Sex And Relationships. Her writing credits include personal essays in the Washington Post, American Baby, the Bark, USAToday, Literary Mama and the Christian Science Monitor as well as articles on a variety of subjects for The Atlantic, AARP Bulletin, Parenting, National Geographic Traveler, BrainWork and American Baby magazines.

More From Kayt Sukel

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts


July 25 • 11:07 AM

The West’s Groundwater Is Being Sucked Dry

Scientists were stunned to discover just how much groundwater has been lost from beneath the Colorado River over the past 10 years.


July 25 • 10:00 AM

Shelf Help: New Book Reviews in 100 Words or Less

What you need to know about Bad Feminist, XL Love, and The Birth of Korean Cool.



July 25 • 8:00 AM

The Consequences of Curing Childhood Cancer

The majority of American children with cancer will be cured, but it may leave them unable to have children of their own. Should preserving fertility in cancer survivors be a research priority?


July 25 • 6:00 AM

Men Find Caring, Understanding Responses Sexy. Women, Not So Much

For women looking to attract a man, there are advantages to being a caring conversationalist. But new research finds it doesn’t work the other way around.


July 25 • 4:00 AM

Arizona’s Double-Talk on Execution and Torture

The state is certain that Joseph Wood’s death was totally constitutional. But they’re looking into it.


July 24 • 4:00 PM

Overweight Americans Have the Lowest Risk of Premature Death

Why do we use the term “normal weight” when talking about BMI? What’s presented as normal certainly isn’t the norm, and it may not even be what’s most healthy.


July 24 • 2:00 PM

California’s Lax Policing of the Fracking Industry Has Put the Drought-Stricken State in a Terrible Situation

The state’s drought has forced farmers to rely on groundwater, even as aquifers have been intentionally polluted due to exemptions for the oil industry.


July 24 • 12:00 PM

What’s in a Name? The Problem With Washington’s Football Team

A senior advisor to the National Congress of American Indians once threw an embarrassing themed party that involved headdresses. He regrets that costume now, but knows his experience is one many others can relate to.


July 24 • 11:00 AM

How Wildlife Declines Are Leading to Slavery and Terrorism

As wildlife numbers dwindle, wildlife crimes are rising—and that’s fueling a raft of heinous crimes committed against humans.


July 24 • 10:58 AM

How the Supremes Pick Their Cases—and Why Obamacare Is Safe for Now

The opponents of Obamacare who went one for two in circuit court rulings earlier this week are unlikely to see their cases reach the Supreme Court.



July 24 • 9:48 AM

The People Who Are Scared of Dogs

While more people fear snakes or spiders, with dogs everywhere, cynophobia makes everyday public life a constant challenge.


July 24 • 8:00 AM

Newton’s Needle: On Scientific Self-Experimentation

It is all too easy to treat science as a platform that allows the observer to hover over the messiness of life, unobserved and untouched. But by remembering the role of the body in science, perhaps we humanize it as well.


July 24 • 6:00 AM

Commercializing the Counterculture: How the Summer Music Festival Went Mainstream

With painted Volkswagen buses, talk of “free love,” and other reminders of the Woodstock era replaced by advertising and corporate sponsorships, hippie culture may be dying, but a new subculture—a sort of purgatory between hipster and hippie—is on the rise.


July 24 • 5:00 AM

In Praise of Our Short Attention Spans

Maybe there’s a good reason why it seems like there’s been a decline in our our ability to concentrate for a prolonged period of time.


July 24 • 4:00 AM

How Stereotypes Take Shape

New research from Scotland finds they’re an unfortunate product of the way we process and share information.


July 23 • 4:00 PM

Who Doesn’t Like Atheists?

The Pew Research Center asked Americans of varying religious affiliations how they felt about each other.


July 23 • 2:00 PM

We Need to Start Tracking Patient Harm and Medical Mistakes Now

Top patient-safety experts call on Congress to step in and, among other steps, give the Centers for Disease Control and Prevention wider responsibility for measuring medical mistakes.


July 23 • 12:19 PM

How a CEO’s Fiery Battle Speeches Can Shape Ethical Behavior

CEO war speech might inspire ethical decisions internally and unethical ones among competing companies.


July 23 • 12:00 PM

Why Do We Love the ‘Kim Kardashian: Hollywood’ Game?

It’s easy enough to turn yourself into a virtual celebrity, complete with fame and mansions—but it will likely cost you.


July 23 • 11:49 AM

Modern Technology Still Doesn’t Protect Americans From Deadly Landslides

No landslide monitoring or warning systems are being used to protect vulnerable communities.


July 23 • 10:00 AM

Outing the Death-Drug Distributors

Calling all hackers: It’s time to go Assange on capital punishment.


July 23 • 8:00 AM

The Surprising Appeal of Products That Require Effort to Use

New research finds they enable consumers to re-establish a feeling that they’re in control of their lives.


Follow us


Subscribe Now

The West’s Groundwater Is Being Sucked Dry

Scientists were stunned to discover just how much groundwater has been lost from beneath the Colorado River over the past 10 years.

How Wildlife Declines Are Leading to Slavery and Terrorism

As wildlife numbers dwindle, wildlife crimes are rising—and that's fueling a raft of heinous crimes committed against humans.

How a CEO’s Fiery Battle Speeches Can Shape Ethical Behavior

CEO war speech might inspire ethical decisions internally and unethical ones among competing companies.

Modern Technology Still Doesn’t Protect Americans From Deadly Landslides

No landslide monitoring or warning systems are being used to protect vulnerable communities.

The Link Between Carbs, Gut Microbes, and Colon Cancer

Reduced carb intake among mice protected them from colon cancer.

The Big One

Today, the United States produces less than two percent of the clothing purchased by Americans. In 1990, it produced nearly 50 percent. July/August 2014

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.