Menus Subscribe Search

Follow us


Sociological Images

noah-film

Noah. (Photo: Paramount Pictures)

Why Survey Questions Matter: Blasphemy Edition

• March 19, 2014 • 4:00 PM

Noah. (Photo: Paramount Pictures)

“As a Faith Driven Consumer, are you satisfied with a Biblically themed movie — designed to appeal to you — which replaces the Bible’s core message with one created by Hollywood?”

“How could we get evidence for this?” I often ask students. And the answer, almost always is, “Do a survey.” The word survey has magical power; anything designated by that name wears a cloak of infallibility.

“Survey just means asking a bunch of people a bunch of questions,” I’ll say. “Whether it has any value depends on how good the bunch of people is and how good the questions are.” My hope is that a few examples of bad sampling and bad questions will demystify.

For example, Variety:

115

Here’s the lede:

Despite its Biblical inspiration, Paramount’s upcoming “Noah” may face some rough seas with religious audiences, according to a new survey by Faith Driven Consumers.

The data to confirm that idea:

The religious organization found in a survey that 98% of its supporters were not “satisfied” with Hollywood’s take on religious stories such as “Noah,” which focuses on Biblical figure Noah.

The sample:

Faith Driven Consumers surveyed its supporters over several days and based the results on a collected 5,000+ responses.

And (I’m saving the best for last) here’s the crucial survey question:

As a Faith Driven Consumer, are you satisfied with a Biblically themed movie — designed to appeal to you — which replaces the Bible’s core message with one created by Hollywood?

As if the part about “replacing the Bible’s core message” weren’t enough, the item reminds the respondent of her or his identity as a Faith Driven Consumer. It does make you wonder about that two percent who either were fine with the Hollywood* message or didn’t know.

You can’t really fault Faith Driven Consumer too much for this shoddy “research.” They’re not in business to find the sociological facts. What’s appalling is that Variety accepts it at face value and without comment.

*The director of Noah is Daniel Aronofsky; the script is credited to him and Ari Handel. For the Faith Driven Consumer, “Hollywood” may carry connotations in addition to that of industry and location—perhaps something similar to “New York sense of humor” in this clip  from The West Wing (the whole six minutes is worth watching, but you’ll get the idea if you push the pointer to 2:20 or so and watch for the next 45 seconds).


This post originally appeared on Sociological Images, a Pacific Standard partner site.

Jay Livingston
Jay Livingston is the chair of the Sociology Department at Montclair State University.

More From Jay Livingston

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts

October 30 • 5:08 AM

We’re Not So Great at Rejecting Each Other

And it’s probably something we should work on.


October 30 • 4:00 AM

He’s Definitely a Liberal—Just Check Out His Brain Scan

New research finds political ideology can be easily determined by examining how one’s brain reacts to disgusting images.


October 29 • 4:00 PM

Should We Prosecute Climate Change Protesters Who Break the Law?

A conversation with Bristol County, Massachusetts, District Attorney Sam Sutter, who dropped steep charges against two climate change protesters.


October 29 • 2:23 PM

Innovation Geography: The Beginning of the End for Silicon Valley

Will a lack of affordable housing hinder the growth of creative start-ups?


October 29 • 2:00 PM

Trapped in the Tobacco Debt Trap

A refinance of Niagara County, New York’s tobacco bonds was good news—but for investors, not taxpayers.


October 29 • 12:00 PM

Purity and Self-Mutilation in Thailand

During the nine-day Phuket Vegetarian Festival, a group of chosen ones known as the mah song torture themselves in order to redirect bad luck and misfortune away from their communities and ensure a year of prosperity.


October 29 • 10:00 AM

Can Proposition 47 Solve California’s Problem With Mass Incarceration?

Reducing penalties for low-level felonies could be the next step in rolling back draconian sentencing laws and addressing the criminal justice system’s long legacy of racism.


October 29 • 9:00 AM

Chronic Fatigue Syndrome and the Brain

Neuroscientists find less—but potentially stronger—white matter in the brains of patients with CFS.


October 29 • 8:00 AM

America’s Bathrooms Are a Total Failure

No matter which American bathroom is crowned in this year’s America’s Best Restroom contest, it will still have a host of terrible flaws.



October 29 • 6:00 AM

Tell Us What You Really Think

In politics, are we always just looking out for No. 1?


October 29 • 4:00 AM

Racial Resentment Drives Tea Party Membership

New research finds a strong link between tea party membership and anti-black feelings.


October 28 • 4:00 PM

The New Health App on Apple’s iOS 8 Is Literally Dangerous

Design isn’t neutral. Design is a picture of inequality, of systems of power, and domination both subtle and not. Apple should know that.


October 28 • 2:00 PM

And You Thought Your Credit Card Debt Was Bad

In Niagara County, New York, leaders took on 40-year debt to pay for short-term stuff, a case study in the perverse incentives tobacco bonds create.



October 28 • 10:00 AM

How Valuable Is It to Cure a Disease?

It depends on the disease—for some, breast cancer and AIDS for example, non-curative therapy that can extend life a little or a lot is considered invaluable. For hepatitis C, it seems that society and the insurance industry have decided that curative therapy simply costs too much.


October 28 • 8:00 AM

Can We Read Our Way Out of Sadness?

How books can help save lives.



October 28 • 6:15 AM

Incumbents, Pray for Rain

Come next Tuesday, rain could push voters toward safer, more predictable candidates.


October 28 • 6:00 AM

Why Women Are Such a Minority in Elected Office

The obvious answers aren’t necessarily the most accurate. Here, five studies help clear up the gender disparity in politics.


October 28 • 4:00 AM

The Study of Science Leads to Leftward Leanings

Researchers report the scientific ethos tends to produce a mindset that favors liberal political positions.


October 28 • 2:00 AM

Who Funded That? The Names and Numbers Behind the Research in Our Latest Issue

This list includes studies cited in our pages that received funding from a source other than the researchers’ home institutions. Only principal or corresponding authors are listed.


October 27 • 4:00 PM

School Shootings: What’s Different About Europe?

There may be a lot of issues at play, but it’s undeniable that the ease of access to guns in the United States is a major contributing factor to our ongoing school shooting crisis.


October 27 • 2:00 PM

The Best Investigative Reporting on Campaign Finance Since 2012

From dark money to a mysterious super PAC donor, here are a few of the best investigations of money in politics since the last elections.


October 27 • 12:00 PM

Could Economics Benefit From Computer Science Thinking?

Computational complexity could offer new insight into old ideas in biology and, yes, even the dismal science.


Follow us


We’re Not So Great at Rejecting Each Other

And it's probably something we should work on.

Chronic Fatigue Syndrome and the Brain

Neuroscientists find less—but potentially stronger—white matter in the brains of patients with CFS.

Incumbents, Pray for Rain

Come next Tuesday, rain could push voters toward safer, more predictable candidates.

Could Economics Benefit From Computer Science Thinking?

Computational complexity could offer new insight into old ideas in biology and, yes, even the dismal science.

Politicians Really Aren’t Better Decision Makers

Politicians took part in a classic choice experiment but failed to do better than the rest of us.

The Big One

One town, Champlain, New York, was the source of nearly half the scams targeting small businesses in the United States last year. November/December 2014

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.