Menus Subscribe Search

Follow us


academic-journals

Academic journals in a university library. (PHOTO: DOTSHOCK/SHUTTERSTOCK)

U.S. Excels at Producing Suspect Behavioral Science

• August 26, 2013 • 12:00 PM

Academic journals in a university library. (PHOTO: DOTSHOCK/SHUTTERSTOCK)

The mythbusters of academe take on soft science from the United States, and find a propensity toward aggrandizement.

Alexis de Tocqueville, bless his oft-cited little pen, first suggested “American exceptionalism,” the idea that the U.S. of A. is qualitatively different from other nations. AE has had some tough slogging of late, with commentators on the left and center arguing it never or no longer exists, while those on the right wring their hands over American erosion. Even our scandals du jour generate another round of navel gazing.

But a new study in the Proceedings of the National Academy of Sciences looking at scientific research offers a backhanded endorsement of American exceptionalism, arguing that in the publish-or-perish world of so-called soft science (I hear the gritting of teeth already), results from U.S.-based research tend to be more extreme in their results. When there can be some ambiguity in the findings, we’re No. 1!

While on its face reporting significant findings more often could seem like a good thing, it’s at best statistically suspect (especially because it’s concentrated in behavioral fields) and worst scientifically suspect. The University of Edinburgh’s Daniele Fanelli and Stanford’s John P.A. Ioannidis suggest this “U.S. effect” can be traced to two independent sources: the nature of behavioral research and the tenor of American academe. Both suggestions are damning, even if they’re not exactly new.

The publication of false, exaggerated, and falsified findings is believed to be more common in research fields where replication is difficult, theories are less clear, and methods are less standardized, because researchers have more “degrees of freedom” to produce the results they expect. Behavioral methodologies, in particular, have been considered at higher risk of bias at least since the 1970s. The intuitive assumption that theoretical and methodological “softness” might increase the prevalence of expectation biases is supported by direct studies of the literature, which suggest that the proportion of papers reporting “positive” outcomes increases moving from the physical to the medical and social sciences and, independent of discipline, is higher among social and behavioral studies on people, compared with nonbehavioral studies and studies on nonhuman behavior.

And on the U.S. in particular?

Many concerns have been expressed, in particular, for the “publish-or-perish” philosophy that has long characterized research in the United States and is increasingly taken up in other countries. Such concerns are increasingly supported by evidence. Researchers working the United States report, in surveys, higher pressures than those in most other countries. At least two independent meta-analyses, one in economics and one in genetic association studies, had noted signs of a larger publication bias among papers from the United States (or North America). The proportion of reported positive results has increased in recent years in most social and biomedical sciences and is greater in US studies, particularly among the most academically productive states.

And let’s not even open Pandora’s box on that whole WEIRD thing….

The messengers of this latest missive are old, and dab, hands at ferreting out chronic research problems. Fanelli has been publicly critiquing flaws in the sociology of science, soft and hard, for more than half a decade, while Ioannidis is a long-standing and high-profile critic of research shortcomings. Papers like “Why Most Discovered True Associations Are Inflated” and “Why Most Published Research Findings Are False” have emanated from his pen for some time. While they may not be subtle, both gentlemen are credible, not cranks, who want to save science from its own expectancy bias.

As Ioannidis told The Atlantic: “Science is a noble endeavor, but it’s also a low-yield endeavor. I’m not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life. We should be very comfortable with that fact.”

In this latest piece, they tracked down 82 recent meta-analyses (a sort of study of studies) in genetics and in psychiatry, the former chosen as an essentially quantitative field and the latter a qualitative one. They then compared how the individual findings in each of the studies that were contained in the meta-analysis—there were 1,172 in all—deviated from the grouped results.

Between the biological and the behavioral studies, the behavioral ones were “significantly more likely to report extreme effects” as a whole, while behavioral results from the U.S. in turn were much more likely to exceed the expectations of their original hypothesis—especially when there were accompanying biological parameters to act as a check.

The biological studies, on the other hand, did not share this propensity and deviated mostly due to sampling errors and showed greater issues when the number of subjects or samples in the study was small, itself a known bias in research. Oddly enough (or suspiciously enough), small-study effects were less pronounced in work from the U.S.

The researchers don’t suggest any intentional agency on the part of their U.S. colleagues, but they do suspect there’s a flawed system at play, in which the researchers’ expectations are amplified by their field’s diverse and often non-standardized methodologies and complex subject matter (which Daniele and Ioannidis term its “level of softness”).

“Our preferred hypothesis is derived from the fact that researchers in the United States have been exposed for a longer time than those in other countries to an unfortunate combination of pressures to publish and winner-takes-all system of rewards,” they write. “This condition is believed to push researchers into either producing many results and then only publishing the most impressive ones, or to make the best of what they got by making them seem as important as possible, through post hoc analyses, rehypothesizing, and other more or less questionable practices.”

Being good academicians themselves, the duo are careful not to overstate their own findings, arguing that what they found definitively applies only to what they found, and may not be broadly applicable—although it probably is.

While the U.S. was the victim of this particular blast, its unwelcome exceptionalism is probably merited only through how it fails, not that it fails. The researchers suggest that given enough statistical firepower, they could find biases peculiar to other countries’ research, much as it’s already been identified in some research from Asia—which in turn may be generated by trying to turn some Western heads by reporting spectacular results.

Michael Todd
Most of Michael Todd's career has been spent in newspaper journalism, ranging from papers in the Marshall Islands to tiny California farming communities. Before joining the publishing arm of the Miller-McCune Center, he was managing editor of the national magazine Hispanic Business.

More From Michael Todd

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts

December 19 • 4:00 PM

How a Drug Policy Reform Organization Thinks of the Children

This valuable, newly updated resource for parents is based in the real world.


December 19 • 2:00 PM

Where Did the Ouija Board Come From?

It wasn’t just a toy.


December 19 • 12:00 PM

Social Scientists Can Do More to Eradicate Racial Oppression

Using our knowledge of social systems, all social scientists—black or white, race scholar or not—have an opportunity to challenge white privilege.


December 19 • 10:17 AM

How Scientists Contribute to Bad Science Reporting

By not taking university press officers and research press releases seriously, scientists are often complicit in the media falsehoods they so often deride.


December 19 • 10:00 AM

Pentecostalism in West Africa: A Boon or Barrier to Disease?

How has Ghana stayed Ebola-free despite being at high risk for infection? A look at their American-style Pentecostalism, a religion that threatens to do more harm than good.


December 19 • 8:00 AM

Don’t Text and Drive—Especially If You’re Old

A new study shows that texting while driving becomes even more dangerous with age.


December 19 • 6:12 AM

All That ‘Call of Duty’ With Your Friends Has Not Made You a More Violent Person

But all that solo Call of Duty has.


December 19 • 4:00 AM

Food for Thought: WIC Works

New research finds participation in the federal WIC program, which subsidizes healthy foods for young children, is linked with stronger cognitive development and higher test scores.


December 18 • 4:00 PM

How I Navigated Life as a Newly Sober Mom

Saying “no” to my kids was harder than saying “no” to alcohol. But for their sake and mine, I had to learn to put myself first sometimes.


December 18 • 2:00 PM

Women in Apocalyptic Fiction Shaving Their Armpits

Because our interest in realism apparently only goes so far.


December 18 • 12:00 PM

The Paradox of Choice, 10 Years Later

Paul Hiebert talks to psychologist Barry Schwartz about how modern trends—social media, FOMO, customer review sites—fit in with arguments he made a decade ago in his highly influential book, The Paradox of Choice: Why More Is Less.


December 18 • 10:00 AM

What It’s Like to Spend a Few Hours in the Church of Scientology

Wrestling with thetans, attempting to unlock a memory bank, and a personality test seemingly aimed at people with depression. This is Scientology’s “dissemination drill” for potential new members.


December 18 • 8:00 AM

Gendering #BlackLivesMatter: A Feminist Perspective

Black men are stereotyped as violent, while black women are rendered invisible. Here’s why the gendering of black lives matters.


December 18 • 7:06 AM

Apparently You Can Bring Your Religion to Work

New research says offices that encourage talk of religion actually make for happier workplaces.


December 18 • 6:00 AM

The Very Weak and Complicated Links Between Mental Illness and Gun Violence

Vanderbilt University’s Jonathan Metzl and Kenneth MacLeish address our anxieties and correct our assumptions.


December 18 • 4:00 AM

Should Movies Be Rated RD for Reckless Driving?

A new study finds a link between watching films featuring reckless driving and engaging in similar behavior years later.


December 17 • 4:00 PM

How to Run a Drug Dealing Network in Prison

People tend not to hear about the prison drug dealing operations that succeed. Substance.com asks a veteran of the game to explain his system.


December 17 • 2:00 PM

Gender Segregation of Toys Is on the Rise

Charting the use of “toys for boys” and “toys for girls” in American English.


December 17 • 12:41 PM

Why the College Football Playoff Is Terrible But Better Than Before

The sample size is still embarrassingly small, but at least there’s less room for the availability cascade.


December 17 • 11:06 AM

Canadian Kids Have a Serious Smoking Problem

Bootleg cigarette sales could be leading Canadian teens to more serious drugs, a recent study finds.


December 17 • 10:37 AM

A Public Lynching in Sproul Plaza

When photographs of lynching victims showed up on a hallowed site of democracy in action, a provocation was issued—but to whom, by whom, and why?


December 17 • 8:00 AM

What Was the Job?

This was the year the job broke, the year we accepted a re-interpretation of its fundamental bargain and bought in to the push to get us to all work for ourselves rather than each other.


December 17 • 6:00 AM

White Kids Will Be Kids

Even the “good” kids—bound for college, upwardly mobile—sometimes break the law. The difference? They don’t have much to fear. A professor of race and social movements reflects on her teenage years and faces some uncomfortable realities.



December 16 • 4:00 PM

How Fear of Occupy Wall Street Undermined the Red Cross’ Sandy Relief Effort

Red Cross responders say there was a ban on working with the widely praised Occupy Sandy relief group because it was seen as politically unpalatable.


Follow us


Don’t Text and Drive—Especially If You’re Old

A new study shows that texting while driving becomes even more dangerous with age.

Apparently You Can Bring Your Religion to Work

New research says offices that encourage talk of religion actually make for happier workplaces.

Canadian Kids Have a Serious Smoking Problem

Bootleg cigarette sales could be leading Canadian teens to more serious drugs, a recent study finds.

The Hidden Psychology of the Home Ref

That old myth of home field bias isn’t a myth at all; it’s a statistical fact.

The Big One

One in two United States senators and two in five House members who left office between 1998 and 2004 became lobbyists. November/December 2014

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.