Menus Subscribe Search
(ILLUSTRATION: MÁGOZ)

(ILLUSTRATION: MÁGOZ)

Replicate This

• February 26, 2013 • 4:00 AM

(ILLUSTRATION: MÁGOZ)

Do classic psychological studies published in high-profile journals hold up? The Reproducibility Project aims to find out.

There are few psychological effects better known—or more widely accepted—in academic halls than what is called semantic priming. Show a person a simple stimulus, something as unremarkable as a photograph of a cat. Let some time pass, then ask that same person to list as many words as possible that start with the letter c. This person is more likely not only to come up with the word cat, but to mention catlike animals such as cougars and cheetahs, because he was initially primed with that one little kitty cat.

Priming’s reach, of course, stretches far beyond cognitive tests. Therapists use it to help treat patients with depression during therapy sessions. Advertisers count on commercials to prime us to buy key brands during our trips to the mall or the grocery store. Priming is considered an underlying mechanism in stereotyping. And the word has become part of our cultural lexicon, too. We talk about how we are “primed” to feel, to want, to need, to talk. Priming is everywhere.

And yet, many of the classic studies that led us to our current understanding of priming have never been replicated. In fact, the few attempts to reproduce the results that we have taken at face value for so long have failed. In late 2012, that led Daniel Kahneman, noted Princeton University psychologist and author of the best-selling book Thinking Fast and Slow, to write an open e-mail to the entire priming-research community. He wrote, “Your field is now the poster child for doubts about the integrity of psychological research. Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published.” Kahneman’s solution? A new research protocol whereby cooperating labs attempt to check and replicate each other’s studies. This is the only way, he argues, to separate the scientific wheat from the chaff.

But accuracy and integrity issues are not limited to studies about semantic priming. They plague the whole psychological community. Research replication, an essential feature of good science—the element that allows truth to shine through the experimental brume—has simply not been a priority in today’s “publish or perish” climate. And we’re now learning that many well-publicized studies can’t be replicated. (This may be due to, say, incorrect or inappropriate analysis of results, or a sample size that is too small.) When studies can be reproduced, there is little incentive for scientists to do so.

“Journals are geared toward publishing new stuff, and that new stuff tends to be overwhelmingly positive results,” says Eric Eich, editor of the journal Psychological Science. “Discovery work, across the board, tends to be valued higher than doing confirmation or replication work.”

So with reputations hanging in the balance, what can be done?

That’s the question Brian Nosek, a psychologist at the University of Virginia, has been pondering. Last year, he launched the crowdsourced Reproducibility Project, with the express mission of replicating psychological studies published in high-profile journals. “Academic science is open, transparent—people are supposed to be able to see the evidence and basis for different claims and then evaluate them,” Nosek tells me. “Science values truth above all else.”

The first step: to find enough psychologists willing to forgo the prestige of discovery work. Nosek reached out to his colleagues across the country—psychologists who, like him, had been decrying the lack of replication in the field at conferences and meetings over the past decade. He challenged each of them to try to reproduce a single study from a sample of those published in three eminent psychological journals. By spreading the work around, and mitigating the difficulties and costs involved with replication, Nosek argued, the field could finally get an idea of what social psychology’s reproducibility rate really is.

The project has been met with overwhelming praise—publicly, at least. Nosek’s original recruitment e-mail went viral, reaching a larger audience than he ever imagined. Now more than 100 scientists across more than 40 global institutions have joined the reproducibility mission, using their own laboratory resources, and the project’s guidelines, in attempts to replicate studies. About 20 replication attempts have already been completed.

Georg Jahn, a psychologist at the University of Greifswald, in Germany, recently replicated a 2008 study that looked at the importance of attention when learning the associations between adjacent and nonadjacent items—a skill that has implications for the way people learn language and grammar. Jahn says the replication was very straightforward, and the results confirmed the original study’s findings.

But reproducing studies, and determining whether original results hold up, is not always so clear-cut. Privately, scientists have voiced concerns about what counts as a true replication. What if the exact same materials or methods are not used? What if the study is run in a setting with slightly more—or less—controls? The devil, as they say, is in the details.

Michael Frank, a developmental psychologist at Stanford University, ran across these issues as he and his students worked on the replications of several survey-type studies. One was a paper titled “Why People Are Reluctant to Tempt Fate.” (pdf) Lead author Jane Risen, at the University of Chicago, and her coauthors documented that individuals, reacting to a series of “what if?” scenarios, responded that it was bad luck to tempt fate—even when they didn’t believe in fate.

Risen’s goal was to find out how and why people can believe things they know are false. Frank’s preliminary results suggest that the magical thinking described in the original paper was not evident in the new study. Risen, though, believes Frank’s students’ work was not a real replication. The Stanford group ran the study on the general population instead of on a subset of students—and did so using the Internet, whereas the original study was conducted in person. “Because the study involved participants imagining themselves being called on in a college class, it was important that it be run with student participants who could relate to the story,” Risen said.

Nosek is quick to tell me that a study could fail to replicate an original result for many reasons: study methods may be just different enough, or a study’s sample not quite large enough to be statistically valid. He admits that he and his project partners are learning as they go—and before making any declarations about a researcher’s success or failure to reproduce a study, they are evaluating every project on a case-by-case basis.

Accuracy is key. If they get a low reproducibility rate, the outcomes, obviously, can have weighty consequences. Donors may be less inclined to fund studies in the field, Nosek explains, or people may lose their trust in psychology altogether.

Kayt Sukel
Kayt Sukel is the author of Dirty Minds: How Our Brains Influence Love, Sex And Relationships. Her writing credits include personal essays in the Washington Post, American Baby, the Bark, USAToday, Literary Mama and the Christian Science Monitor as well as articles on a variety of subjects for The Atlantic, AARP Bulletin, Parenting, National Geographic Traveler, BrainWork and American Baby magazines.

More From Kayt Sukel

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts

August 29 • 4:00 PM

The Hidden Costs of Tobacco Debt

Even when taxpayers aren’t explicitly on the hook, tobacco bonds can cost states and local governments money. Here’s how.


August 29 • 2:00 PM

Why Don’t Men and Women Wear the Same Gender-Neutral Bathing Suits?

They used to in the 1920s.


August 29 • 11:48 AM

Your Brain Decides Whether to Trust Someone in Milliseconds

We can determine trustworthiness even when we’re only subliminally aware of the other person.


August 29 • 10:00 AM

True Darwinism Is All About Chance

Though the rich sometimes forget, Darwin knew that nature frequently rolls the dice.


August 29 • 8:00 AM

Why Our Molecular Make-Up Can’t Explain Who We Are

Our genes only tell a portion of the story.


August 29 • 6:00 AM

Strange Situations: Attachment Theory and Sexual Assault on College Campuses

When college women leave home, does attachment behavior make them more vulnerable to campus rape?


August 29 • 4:00 AM

Forgive Your Philandering Partner—and Pay the Price

New research finds people who forgive an unfaithful romantic partner are considered weaker and less competent than those who ended the relationship.


August 28 • 4:00 PM

Some Natural-Looking Zoo Exhibits May Be Even Worse Than the Old Concrete Ones

They’re often designed for you, the paying visitor, and not the animals who have to inhabit them.


August 28 • 2:00 PM

What I Learned From Debating Science With Trolls

“Don’t feed the trolls” is sound advice, but occasionally ignoring it can lead to rewards.


August 28 • 12:00 PM

The Ice Bucket Challenge’s Meme Money

The ALS Association has raised nearly $100 million over the past month, 50 times what it raised in the same period last year. How will that money be spent, and how can non-profit executives make a windfall last?


August 28 • 11:56 AM

Outlawing Water Conflict: California Legislators Confront Risky Groundwater Loophole

California, where ambitious agriculture sucks up 80 percent of the state’s developed water, is no stranger to water wrangles. Now one of the worst droughts in state history is pushing legislators to reckon with its unwieldy water laws, especially one major oversight: California has been the only Western state without groundwater regulation—but now that looks set to change.


August 28 • 11:38 AM

Young, Undocumented, and Invisible

While young migrant workers struggle under poor working conditions, U.S. policy has done little to help.


August 28 • 10:00 AM

The Five Words You Never Want to Hear From Your Doctor

“Sometimes people just get pains.”


August 28 • 8:00 AM

Why I’m Not Sharing My Coke

Andy Warhol, algorithms, and a bunch of popular names printed on soda cans.


August 28 • 6:00 AM

Can Outdoor Art Revitalize Outdoor Advertising?

That art you’ve been seeing at bus stations and billboards—it’s serving a purpose beyond just promoting local museums.


August 28 • 4:00 AM

Linguistic Analysis Reveals Research Fraud

An examination of papers by the discredited Diederik Stapel finds linguistic differences between his legitimate and fraudulent studies.


August 28 • 2:00 AM

Poverty and Geography: The Myth of Racial Segregation

Migration, regardless of race, ethnicity, gender, or sexuality (not to mention class), can be a poverty-buster.


August 27 • 4:00 PM

The ‘Non-Lethal’ Flash-Bang Grenades Used in Ferguson Can Actually Be Quite Lethal

A journalist says he was singed by a flash-bang fired by St. Louis County police trying to disperse a crowd, raising questions about how to use these military-style devices safely and appropriately.


August 27 • 2:00 PM

Do Better Looking People Have Better Personalities Too?

An experiment on users of the dating site OKCupid found that members judge both looks and personality by looks alone.


August 27 • 12:00 PM

Love Can Make You Stronger

A new study links oxytocin, the hormone most commonly associated with social bonding, and the one that your body produces during an orgasm, with muscle regeneration.


August 27 • 11:05 AM

Education, Interrupted

When it comes to educational access, young Syrian refugees are becoming a “lost generation.”


August 27 • 9:47 AM

No, Smartphone-Loss Anxiety Disorder Isn’t Real

But people are anxious about losing their phones, even if they don’t do much to protect them.


August 27 • 8:00 AM

A Skeptic Meets a Psychic: When You Can See Into the Future, How Do You Handle Uncertainty?

For all the crystal balls and beaded doorways, some psychics provide a useful, non-paranormal service. The best ones—they give good advice.


August 27 • 6:00 AM

Speaking Eyebrow: Your Face Is Saying More Than You Think

Our involuntary gestures take on different “accents” depending on our cultural background.


August 27 • 4:00 AM

The Politics of Anti-NIMBYism and Addressing Housing Affordability

Respected expert economists like Paul Krugman and Edward Glaeser are confusing readers with their poor grasp of demography.


Follow us


Subscribe Now

Your Brain Decides Whether to Trust Someone in Milliseconds

We can determine trustworthiness even when we’re only subliminally aware of the other person.

Young, Undocumented, and Invisible

While young migrant workers struggle under poor working conditions, U.S. policy has done little to help.

Education, Interrupted

When it comes to educational access, young Syrian refugees are becoming a “lost generation.”

No, Smartphone-Loss Anxiety Disorder Isn’t Real

But people are anxious about losing their phones, even if they don’t do much to protect them.

Being a Couch Potato: Not So Bad After All?

For those who feel guilty about watching TV, a new study provides redemption.

The Big One

One in two full-time American fast-food workers' families are enrolled in public assistance programs, at a cost of $7 billion per year. July/August 2014 fast-food-big-one

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.