Menus Subscribe Search

Sociological Images

noah-film

Why Survey Questions Matter: Blasphemy Edition

• March 19, 2014 • 4:00 PM

Noah. (Photo: Paramount Pictures)

“As a Faith Driven Consumer, are you satisfied with a Biblically themed movie — designed to appeal to you — which replaces the Bible’s core message with one created by Hollywood?”

“How could we get evidence for this?” I often ask students. And the answer, almost always is, “Do a survey.” The word survey has magical power; anything designated by that name wears a cloak of infallibility.

“Survey just means asking a bunch of people a bunch of questions,” I’ll say. “Whether it has any value depends on how good the bunch of people is and how good the questions are.” My hope is that a few examples of bad sampling and bad questions will demystify.

For example, Variety:

115

Here’s the lede:

Despite its Biblical inspiration, Paramount’s upcoming “Noah” may face some rough seas with religious audiences, according to a new survey by Faith Driven Consumers.

The data to confirm that idea:

The religious organization found in a survey that 98% of its supporters were not “satisfied” with Hollywood’s take on religious stories such as “Noah,” which focuses on Biblical figure Noah.

The sample:

Faith Driven Consumers surveyed its supporters over several days and based the results on a collected 5,000+ responses.

And (I’m saving the best for last) here’s the crucial survey question:

As a Faith Driven Consumer, are you satisfied with a Biblically themed movie — designed to appeal to you — which replaces the Bible’s core message with one created by Hollywood?

As if the part about “replacing the Bible’s core message” weren’t enough, the item reminds the respondent of her or his identity as a Faith Driven Consumer. It does make you wonder about that two percent who either were fine with the Hollywood* message or didn’t know.

You can’t really fault Faith Driven Consumer too much for this shoddy “research.” They’re not in business to find the sociological facts. What’s appalling is that Variety accepts it at face value and without comment.

*The director of Noah is Daniel Aronofsky; the script is credited to him and Ari Handel. For the Faith Driven Consumer, “Hollywood” may carry connotations in addition to that of industry and location—perhaps something similar to “New York sense of humor” in this clip  from The West Wing (the whole six minutes is worth watching, but you’ll get the idea if you push the pointer to 2:20 or so and watch for the next 45 seconds).


This post originally appeared on Sociological Images, a Pacific Standard partner site.

Jay Livingston

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts

July 22 • 12:00 PM

On the Destinations of Species

It’s almost always easier to cross international borders if you’re something other than human.


July 22 • 10:51 AM

The Link Between Carbs, Gut Microbes, and Colon Cancer

Reduced carb intake among mice protected them from colon cancer.


July 22 • 10:47 AM

Irrational Choice Theory: The LeBron James Migration From Miami to Cleveland

Return migrants to Cleveland have been coming home in large numbers for quite some time. It makes perfect sense.


July 22 • 9:32 AM

This Time, Scalia Was Right

President Obama’s recess appointments were wrong and, worse, dangerous.


July 22 • 8:00 AM

On Vegas Strip, Blackjack Rule Change Is Sleight of Hand

Casino operators are changing blackjack payouts to give the house an even greater advantage. Is this a sign that Vegas is on its way back from the recession, or that the Strip’s biggest players are trying to squeeze some more cash out of visitors before the well runs dry?


July 22 • 6:00 AM

Label Me Confused

How the words on a bag of food create more questions than answers.


July 22 • 5:07 AM

Doubly Victimized: The Shocking Prevalence of Violence Against Homeless Women

An especially vulnerable population is surveyed by researchers.


July 22 • 4:00 AM

New Evidence That Blacks Are Aging Faster Than Whites

A large study finds American blacks are, biologically, three years older than their white chronological counterparts.



July 21 • 4:00 PM

Do You Have to Learn How to Get High?

All drugs are socially constructed.


July 21 • 2:14 PM

The New Weapon Against Disease-Spreading Insects Is Big Data

Computer models that pinpoint the likely locations of mosquitoes and tsetse flies are helping officials target vector control efforts.


July 21 • 2:00 PM

Why Are Obstetricians Among the Top Billers for Group Psychotherapy in Illinois?

Illinois leads the country in group psychotherapy sessions in Medicare, and some top billers aren’t mental health specialists. The state’s Medicaid program has cracked down, but federal officials have not.



July 21 • 12:00 PM

What Makes You So Smart, MacArthur Genius?

Noah Davis talks to Yoky Matsuoka about youth tennis, wanting to be an airhead, and what it’s like to win a Genius Grant.


July 21 • 11:23 AM

People Are Clueless About Placebos

Doctors know that sometimes the best medicine is no medicine at all. But how do patients feel about getting duped into recovery?


July 21 • 10:00 AM

How Small-D Democratic Should Our Political Parties Be?

We need to decide how primaries should work in this country before they get completely out of hand and the voters are left out entirely.


July 21 • 8:00 AM

No, Walking on All 4 Limbs Is Not a Sign of Human ‘Devolution’

New quantitative analysis reveals that people with Uner Tan Syndrome don’t actually walk like primates at all.


July 21 • 6:00 AM

Sequenced in the U.S.A.: A Desperate Town Hands Over Its DNA

The new American economy in three tablespoons of blood, a Walmart gift card, and a former mill town’s DNA.


July 21 • 5:00 AM

Celebrating Independence: Scenes From 59 Days Around the World

While national identities are often used to separate people, a husband-and-wife Facebook photography project aims to build connections.


July 21 • 4:00 AM

Be a Better Person: Take a Walk in the Park

New research from France finds strangers are more helpful if they’ve just strolled through a natural environment.



July 18 • 4:00 PM

The Litany of Problems With the Pentagon’s Effort to Recover MIAs

A draft inspector general report found that the mission lacks basic metrics for how to do the job—and when to end it.


July 18 • 2:00 PM

Sure, the Jobs Are Back, but We Need a Lot More

We’re back to where we were before the 2008 recession, but there are now 12 million more people in the United States.


July 18 • 12:00 PM

What Are the Benefits of Government-Funded Research?

Congress wants to know.


July 18 • 10:31 AM

Why Didn’t California’s Handheld Phone Ban Reduce Motor Accidents?

Are handheld cell phones as dangerous as they have been made out to be?


Follow us


Subscribe Now

The Link Between Carbs, Gut Microbes, and Colon Cancer

Reduced carb intake among mice protected them from colon cancer.

The New Weapon Against Disease-Spreading Insects Is Big Data

Computer models that pinpoint the likely locations of mosquitoes and tsetse flies are helping officials target vector control efforts.

People Are Clueless About Placebos

Doctors know that sometimes the best medicine is no medicine at all. But how do patients feel about getting duped into recovery?

No, Walking on All 4 Limbs Is Not a Sign of Human ‘Devolution’

New quantitative analysis reveals that people with Uner Tan Syndrome don't actually walk like primates at all.

Why Didn’t California’s Handheld Phone Ban Reduce Motor Accidents?

Are handheld cell phones as dangerous as they have been made out to be?

The Big One

Today, the United States produces less than two percent of the clothing purchased by Americans. In 1990, it produced nearly 50 percent. July/August 2014

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.