Menus Subscribe Search
academic-journals

Academic journals in a university library. (PHOTO: DOTSHOCK/SHUTTERSTOCK)

U.S. Excels at Producing Suspect Behavioral Science

• August 26, 2013 • 12:00 PM

Academic journals in a university library. (PHOTO: DOTSHOCK/SHUTTERSTOCK)

The mythbusters of academe take on soft science from the United States, and find a propensity toward aggrandizement.

Alexis de Tocqueville, bless his oft-cited little pen, first suggested “American exceptionalism,” the idea that the U.S. of A. is qualitatively different from other nations. AE has had some tough slogging of late, with commentators on the left and center arguing it never or no longer exists, while those on the right wring their hands over American erosion. Even our scandals du jour generate another round of navel gazing.

But a new study in the Proceedings of the National Academy of Sciences looking at scientific research offers a backhanded endorsement of American exceptionalism, arguing that in the publish-or-perish world of so-called soft science (I hear the gritting of teeth already), results from U.S.-based research tend to be more extreme in their results. When there can be some ambiguity in the findings, we’re No. 1!

While on its face reporting significant findings more often could seem like a good thing, it’s at best statistically suspect (especially because it’s concentrated in behavioral fields) and worst scientifically suspect. The University of Edinburgh’s Daniele Fanelli and Stanford’s John P.A. Ioannidis suggest this “U.S. effect” can be traced to two independent sources: the nature of behavioral research and the tenor of American academe. Both suggestions are damning, even if they’re not exactly new.

The publication of false, exaggerated, and falsified findings is believed to be more common in research fields where replication is difficult, theories are less clear, and methods are less standardized, because researchers have more “degrees of freedom” to produce the results they expect. Behavioral methodologies, in particular, have been considered at higher risk of bias at least since the 1970s. The intuitive assumption that theoretical and methodological “softness” might increase the prevalence of expectation biases is supported by direct studies of the literature, which suggest that the proportion of papers reporting “positive” outcomes increases moving from the physical to the medical and social sciences and, independent of discipline, is higher among social and behavioral studies on people, compared with nonbehavioral studies and studies on nonhuman behavior.

And on the U.S. in particular?

Many concerns have been expressed, in particular, for the “publish-or-perish” philosophy that has long characterized research in the United States and is increasingly taken up in other countries. Such concerns are increasingly supported by evidence. Researchers working the United States report, in surveys, higher pressures than those in most other countries. At least two independent meta-analyses, one in economics and one in genetic association studies, had noted signs of a larger publication bias among papers from the United States (or North America). The proportion of reported positive results has increased in recent years in most social and biomedical sciences and is greater in US studies, particularly among the most academically productive states.

And let’s not even open Pandora’s box on that whole WEIRD thing….

The messengers of this latest missive are old, and dab, hands at ferreting out chronic research problems. Fanelli has been publicly critiquing flaws in the sociology of science, soft and hard, for more than half a decade, while Ioannidis is a long-standing and high-profile critic of research shortcomings. Papers like “Why Most Discovered True Associations Are Inflated” and “Why Most Published Research Findings Are False” have emanated from his pen for some time. While they may not be subtle, both gentlemen are credible, not cranks, who want to save science from its own expectancy bias.

As Ioannidis told The Atlantic: “Science is a noble endeavor, but it’s also a low-yield endeavor. I’m not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life. We should be very comfortable with that fact.”

In this latest piece, they tracked down 82 recent meta-analyses (a sort of study of studies) in genetics and in psychiatry, the former chosen as an essentially quantitative field and the latter a qualitative one. They then compared how the individual findings in each of the studies that were contained in the meta-analysis—there were 1,172 in all—deviated from the grouped results.

Between the biological and the behavioral studies, the behavioral ones were “significantly more likely to report extreme effects” as a whole, while behavioral results from the U.S. in turn were much more likely to exceed the expectations of their original hypothesis—especially when there were accompanying biological parameters to act as a check.

The biological studies, on the other hand, did not share this propensity and deviated mostly due to sampling errors and showed greater issues when the number of subjects or samples in the study was small, itself a known bias in research. Oddly enough (or suspiciously enough), small-study effects were less pronounced in work from the U.S.

The researchers don’t suggest any intentional agency on the part of their U.S. colleagues, but they do suspect there’s a flawed system at play, in which the researchers’ expectations are amplified by their field’s diverse and often non-standardized methodologies and complex subject matter (which Daniele and Ioannidis term its “level of softness”).

“Our preferred hypothesis is derived from the fact that researchers in the United States have been exposed for a longer time than those in other countries to an unfortunate combination of pressures to publish and winner-takes-all system of rewards,” they write. “This condition is believed to push researchers into either producing many results and then only publishing the most impressive ones, or to make the best of what they got by making them seem as important as possible, through post hoc analyses, rehypothesizing, and other more or less questionable practices.”

Being good academicians themselves, the duo are careful not to overstate their own findings, arguing that what they found definitively applies only to what they found, and may not be broadly applicable—although it probably is.

While the U.S. was the victim of this particular blast, its unwelcome exceptionalism is probably merited only through how it fails, not that it fails. The researchers suggest that given enough statistical firepower, they could find biases peculiar to other countries’ research, much as it’s already been identified in some research from Asia—which in turn may be generated by trying to turn some Western heads by reporting spectacular results.

Michael Todd
Most of Michael Todd's career has been spent in newspaper journalism, ranging from papers in the Marshall Islands to tiny California farming communities. Before joining the publishing arm of the Miller-McCune Center, he was managing editor of the national magazine Hispanic Business.

More From Michael Todd

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts

August 29 • 4:00 PM

The Hidden Costs of Tobacco Debt

Even when taxpayers aren’t explicitly on the hook, tobacco bonds can cost states and local governments money. Here’s how.


August 29 • 2:00 PM

Why Don’t Men and Women Wear the Same Gender-Neutral Bathing Suits?

They used to in the 1920s.


August 29 • 11:48 AM

Your Brain Decides Whether to Trust Someone in Milliseconds

We can determine trustworthiness even when we’re only subliminally aware of the other person.


August 29 • 10:00 AM

True Darwinism Is All About Chance

Though the rich sometimes forget, Darwin knew that nature frequently rolls the dice.


August 29 • 8:00 AM

Why Our Molecular Make-Up Can’t Explain Who We Are

Our genes only tell a portion of the story.


August 29 • 6:00 AM

Strange Situations: Attachment Theory and Sexual Assault on College Campuses

When college women leave home, does attachment behavior make them more vulnerable to campus rape?


August 29 • 4:00 AM

Forgive Your Philandering Partner—and Pay the Price

New research finds people who forgive an unfaithful romantic partner are considered weaker and less competent than those who ended the relationship.


August 28 • 4:00 PM

Some Natural-Looking Zoo Exhibits May Be Even Worse Than the Old Concrete Ones

They’re often designed for you, the paying visitor, and not the animals who have to inhabit them.


August 28 • 2:00 PM

What I Learned From Debating Science With Trolls

“Don’t feed the trolls” is sound advice, but occasionally ignoring it can lead to rewards.


August 28 • 12:00 PM

The Ice Bucket Challenge’s Meme Money

The ALS Association has raised nearly $100 million over the past month, 50 times what it raised in the same period last year. How will that money be spent, and how can non-profit executives make a windfall last?


August 28 • 11:56 AM

Outlawing Water Conflict: California Legislators Confront Risky Groundwater Loophole

California, where ambitious agriculture sucks up 80 percent of the state’s developed water, is no stranger to water wrangles. Now one of the worst droughts in state history is pushing legislators to reckon with its unwieldy water laws, especially one major oversight: California has been the only Western state without groundwater regulation—but now that looks set to change.


August 28 • 11:38 AM

Young, Undocumented, and Invisible

While young migrant workers struggle under poor working conditions, U.S. policy has done little to help.


August 28 • 10:00 AM

The Five Words You Never Want to Hear From Your Doctor

“Sometimes people just get pains.”


August 28 • 8:00 AM

Why I’m Not Sharing My Coke

Andy Warhol, algorithms, and a bunch of popular names printed on soda cans.


August 28 • 6:00 AM

Can Outdoor Art Revitalize Outdoor Advertising?

That art you’ve been seeing at bus stations and billboards—it’s serving a purpose beyond just promoting local museums.


August 28 • 4:00 AM

Linguistic Analysis Reveals Research Fraud

An examination of papers by the discredited Diederik Stapel finds linguistic differences between his legitimate and fraudulent studies.


August 28 • 2:00 AM

Poverty and Geography: The Myth of Racial Segregation

Migration, regardless of race, ethnicity, gender, or sexuality (not to mention class), can be a poverty-buster.


August 27 • 4:00 PM

The ‘Non-Lethal’ Flash-Bang Grenades Used in Ferguson Can Actually Be Quite Lethal

A journalist says he was singed by a flash-bang fired by St. Louis County police trying to disperse a crowd, raising questions about how to use these military-style devices safely and appropriately.


August 27 • 2:00 PM

Do Better Looking People Have Better Personalities Too?

An experiment on users of the dating site OKCupid found that members judge both looks and personality by looks alone.


August 27 • 12:00 PM

Love Can Make You Stronger

A new study links oxytocin, the hormone most commonly associated with social bonding, and the one that your body produces during an orgasm, with muscle regeneration.


August 27 • 11:05 AM

Education, Interrupted

When it comes to educational access, young Syrian refugees are becoming a “lost generation.”


August 27 • 9:47 AM

No, Smartphone-Loss Anxiety Disorder Isn’t Real

But people are anxious about losing their phones, even if they don’t do much to protect them.


August 27 • 8:00 AM

A Skeptic Meets a Psychic: When You Can See Into the Future, How Do You Handle Uncertainty?

For all the crystal balls and beaded doorways, some psychics provide a useful, non-paranormal service. The best ones—they give good advice.


August 27 • 6:00 AM

Speaking Eyebrow: Your Face Is Saying More Than You Think

Our involuntary gestures take on different “accents” depending on our cultural background.


August 27 • 4:00 AM

The Politics of Anti-NIMBYism and Addressing Housing Affordability

Respected expert economists like Paul Krugman and Edward Glaeser are confusing readers with their poor grasp of demography.


Follow us


Subscribe Now

Your Brain Decides Whether to Trust Someone in Milliseconds

We can determine trustworthiness even when we’re only subliminally aware of the other person.

Young, Undocumented, and Invisible

While young migrant workers struggle under poor working conditions, U.S. policy has done little to help.

Education, Interrupted

When it comes to educational access, young Syrian refugees are becoming a “lost generation.”

No, Smartphone-Loss Anxiety Disorder Isn’t Real

But people are anxious about losing their phones, even if they don’t do much to protect them.

Being a Couch Potato: Not So Bad After All?

For those who feel guilty about watching TV, a new study provides redemption.

The Big One

One in two full-time American fast-food workers' families are enrolled in public assistance programs, at a cost of $7 billion per year. July/August 2014 fast-food-big-one

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.