Menus Subscribe Search
digital-polling

(ILLUSTRATION: PAVEL IGNATOV/SHUTTERSTOCK)

The Use and Abuse of Polling in American Politics

• September 16, 2013 • 8:00 AM

(ILLUSTRATION: PAVEL IGNATOV/SHUTTERSTOCK)

Last week’s fight over PPP’s decision to hold back the results of a poll highlight how too many pollsters operating in the political sphere take an Ivory Tower attitude, disavowing responsibility for the consequences of their work.

An epic data nerd battle started last Wednesday over Twitter as critics lambasted Public Policy Polling (PPP) for withholding a weekend poll of voters in Pueblo County, Colorado.

The poll asked Pueblo County voters if they supported the legislative recall of State Senator Angela Giron, a moderate Democrat who supported stronger gun control legislation in her state. Giron’s Tuesday recall election and that of State Senate President John Morse, of Colorado Springs, marked the first legislative recalls of state lawmakers in Colorado’s 137-year history.

PPP found 54 percent of Pueblo’s registered voters overall favored Giron’s recall, while just 42 percent opposed it. Stranger yet, a whopping 33 percent of Democratic respondents said that they supported the recall. Consider that 47 percent of Pueblo’s registered voters are Democrats; just 23 percent are Republicans.

“In a district that Barack Obama won by almost 20 points I figured there was no way that could be right,” a post from PPP Director Tom Jensen explained on the firm’s website. Not only that, voters in Pueblo County support expanded background checks for gun buyers—68 to 27 percent—and are split on ammunition limits.

“VERY bad and unscientific practice for @ppppolls to suppress a polling result they didn’t believe/didn’t like.”

PPP suspected a design flaw. “This was the first legislative recall election in Colorado history. There’s been a lot of voter confusion,” Jensen noted. “That finding made me think that respondents may not have understood what they were being asked.” With no time to test a new survey design, they waited for the election results.

Tuesday’s election results matched the unpublished poll results closely: 56 percent for the recall, 44 percent against. On Wednesday morning, PPP posted the weekend poll with an explanation for the delay and analysis. Poll trolling commenced.

PPP is not a public agency, it’s not a non-profit, and it’s not under any obligation to anyone but its clients to release its polls. The group has nevertheless made a policy of being responsive to public interest and shown a commitment to transparency. That’s laudable, not lamentable.

But Nate Silver couldn’t help himself: the much-feted statistician behind FiveThirtyEight threw a punch at the pollsters over Twitter early Wednesday afternoon, announcing, “VERY bad and unscientific practice for @ppppolls to suppress a polling result they didn’t believe/didn’t like.” He linked to the PPP post explaining their delay in releasing the Giron poll results. In his next-day follow-up, Silver accused PPP of an “approach to polling [that] is extremely ad hoc.” He added, “that ad-hockery stems from a lack of appreciation/understanding for the statistical fundamentals behind polling.”

Taking the cue from golden-boy Silver, other commentators and analysts have piled on. Mark Blumenthal, founding editor of Pollster.com and senior polling editor for the Huffington Post, rallied behind Silver by early afternoon Wednesday. He launched into his criticisms of PPP with a quick disclaimer: “I’m late to this #NerdFight.” Thursday brought a particularly pointed piece by Nate Cohn, The New Republic’s self-professed amateur polling analyst. Blumenthal rehashed Cohn’s arguments in a post for the Huffington Post.

It’s worth noting that, with the exception of Blumenthal, few of PPP’s critics have polling chops; journalists and practitioners with experience rallied behind PPP.

Silver criticizing a polling outfit for being reckless and unscientific is like Lindsay Lohan accusing Meryl Streep of lacking work ethic. In November 2011, Silver declared that President Obama had just a 17 percent chance of re-election. I wrote about the many problems with Silver’s piece, statistical and ethical, for the Huffington Post: Silver conceded he used just 17 elections (the post-1944 presidential cycles), discounted the reliability of historical precedent, admitted the inconsistency of economic forecasts, and exposed prevailing methods for predicting electoral outcome through tables and rundowns as no better than hit-or-miss.

In one corner, we have a hitherto respected polling outfit that releases data regularly; in the other, a statistician who strung together a series of untrustworthy variables to land the cover of the New York Times Magazine, justifying the exercise by listing the many problems with his claim in the body of his piece.

Gawker’s Max Read noted, “nate silver is a funny guy to be knocking an organization for lack of transparency!” The Plum Line’s Greg Sargent asked Nate Silver, “are you alleging they ‘suppressed’ the result because they ‘didn’t like’ it?” That “[PPP-Nate Silver] dispute turns out to be pretty narrow,” Brian Beutler of Salon commented. I found myself drawn into the Twitter war on the topic: “If you mixed blue and yellow and got purple, would you announce it before re-examining your experiment?”

It’s refreshing to find that there are still pollsters who self-regulate. Too many pollsters operating in the political sphere take an Ivory Tower attitude, disavowing responsibility for the consequences of their polls and analyses. Their reasoning, as I understand it, is that, so long as they note the margin of error and any other qualifications in fine print, they’re on the up-and-up. On that, they’re wrong.

Pre-election polling can affect election results—even Nate Silver admits as much. We’ve got multiple theories to choose from, each of which carries some weight. Voters may want to choose the winner or vote with their friends, for example. Low-information voters may take a marketplace approach, relying on public opinion to champion the worthiest candidate.

When a pollster puts out a poll with a huge margin of error or a skewed sample frame, most coverage won’t note the fine print, even if the pollster does. The irresponsible pollster publishes the poll, content to slap on a sensational headline or else criticize media for doing so. The responsible pollster recognizes the dangers and acts to ensure that data isn’t misappropriated.

PPP identified a problematic result and held it until they could explain what happened, despite the fact that they could’ve gotten publicity from the polling before the election. They published the poll after the election knowing they might face sustained criticism for having held it back. That’s a commitment to science and public responsibility that we should welcome.

Rebecca Buckwalter-Poza
Rebecca Buckwalter-Poza attended Harvard College and Yale Law School. She has written on law and politics for the Nation, the Atlantic, Politico, the Daily Beast, and CNN, and co-authored James Carville’s 40 More Years. Follow her on Twitter @rpbp.

More From Rebecca Buckwalter-Poza

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts

August 29 • 4:00 PM

The Hidden Costs of Tobacco Debt

Even when taxpayers aren’t explicitly on the hook, tobacco bonds can cost states and local governments money. Here’s how.


August 29 • 2:00 PM

Why Don’t Men and Women Wear the Same Gender-Neutral Bathing Suits?

They used to in the 1920s.


August 29 • 11:48 AM

Your Brain Decides Whether to Trust Someone in Milliseconds

We can determine trustworthiness even when we’re only subliminally aware of the other person.


August 29 • 10:00 AM

True Darwinism Is All About Chance

Though the rich sometimes forget, Darwin knew that nature frequently rolls the dice.


August 29 • 8:00 AM

Why Our Molecular Make-Up Can’t Explain Who We Are

Our genes only tell a portion of the story.


August 29 • 6:00 AM

Strange Situations: Attachment Theory and Sexual Assault on College Campuses

When college women leave home, does attachment behavior make them more vulnerable to campus rape?


August 29 • 4:00 AM

Forgive Your Philandering Partner—and Pay the Price

New research finds people who forgive an unfaithful romantic partner are considered weaker and less competent than those who ended the relationship.


August 28 • 4:00 PM

Some Natural-Looking Zoo Exhibits May Be Even Worse Than the Old Concrete Ones

They’re often designed for you, the paying visitor, and not the animals who have to inhabit them.


August 28 • 2:00 PM

What I Learned From Debating Science With Trolls

“Don’t feed the trolls” is sound advice, but occasionally ignoring it can lead to rewards.


August 28 • 12:00 PM

The Ice Bucket Challenge’s Meme Money

The ALS Association has raised nearly $100 million over the past month, 50 times what it raised in the same period last year. How will that money be spent, and how can non-profit executives make a windfall last?


August 28 • 11:56 AM

Outlawing Water Conflict: California Legislators Confront Risky Groundwater Loophole

California, where ambitious agriculture sucks up 80 percent of the state’s developed water, is no stranger to water wrangles. Now one of the worst droughts in state history is pushing legislators to reckon with its unwieldy water laws, especially one major oversight: California has been the only Western state without groundwater regulation—but now that looks set to change.


August 28 • 11:38 AM

Young, Undocumented, and Invisible

While young migrant workers struggle under poor working conditions, U.S. policy has done little to help.


August 28 • 10:00 AM

The Five Words You Never Want to Hear From Your Doctor

“Sometimes people just get pains.”


August 28 • 8:00 AM

Why I’m Not Sharing My Coke

Andy Warhol, algorithms, and a bunch of popular names printed on soda cans.


August 28 • 6:00 AM

Can Outdoor Art Revitalize Outdoor Advertising?

That art you’ve been seeing at bus stations and billboards—it’s serving a purpose beyond just promoting local museums.


August 28 • 4:00 AM

Linguistic Analysis Reveals Research Fraud

An examination of papers by the discredited Diederik Stapel finds linguistic differences between his legitimate and fraudulent studies.


August 28 • 2:00 AM

Poverty and Geography: The Myth of Racial Segregation

Migration, regardless of race, ethnicity, gender, or sexuality (not to mention class), can be a poverty-buster.


August 27 • 4:00 PM

The ‘Non-Lethal’ Flash-Bang Grenades Used in Ferguson Can Actually Be Quite Lethal

A journalist says he was singed by a flash-bang fired by St. Louis County police trying to disperse a crowd, raising questions about how to use these military-style devices safely and appropriately.


August 27 • 2:00 PM

Do Better Looking People Have Better Personalities Too?

An experiment on users of the dating site OKCupid found that members judge both looks and personality by looks alone.


August 27 • 12:00 PM

Love Can Make You Stronger

A new study links oxytocin, the hormone most commonly associated with social bonding, and the one that your body produces during an orgasm, with muscle regeneration.


August 27 • 11:05 AM

Education, Interrupted

When it comes to educational access, young Syrian refugees are becoming a “lost generation.”


August 27 • 9:47 AM

No, Smartphone-Loss Anxiety Disorder Isn’t Real

But people are anxious about losing their phones, even if they don’t do much to protect them.


August 27 • 8:00 AM

A Skeptic Meets a Psychic: When You Can See Into the Future, How Do You Handle Uncertainty?

For all the crystal balls and beaded doorways, some psychics provide a useful, non-paranormal service. The best ones—they give good advice.


August 27 • 6:00 AM

Speaking Eyebrow: Your Face Is Saying More Than You Think

Our involuntary gestures take on different “accents” depending on our cultural background.


August 27 • 4:00 AM

The Politics of Anti-NIMBYism and Addressing Housing Affordability

Respected expert economists like Paul Krugman and Edward Glaeser are confusing readers with their poor grasp of demography.


Follow us


Subscribe Now

Your Brain Decides Whether to Trust Someone in Milliseconds

We can determine trustworthiness even when we’re only subliminally aware of the other person.

Young, Undocumented, and Invisible

While young migrant workers struggle under poor working conditions, U.S. policy has done little to help.

Education, Interrupted

When it comes to educational access, young Syrian refugees are becoming a “lost generation.”

No, Smartphone-Loss Anxiety Disorder Isn’t Real

But people are anxious about losing their phones, even if they don’t do much to protect them.

Being a Couch Potato: Not So Bad After All?

For those who feel guilty about watching TV, a new study provides redemption.

The Big One

One in two full-time American fast-food workers' families are enrolled in public assistance programs, at a cost of $7 billion per year. July/August 2014 fast-food-big-one

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.