Menus Subscribe Search


The Use and Abuse of Polling in American Politics

• September 16, 2013 • 8:00 AM


Last week’s fight over PPP’s decision to hold back the results of a poll highlight how too many pollsters operating in the political sphere take an Ivory Tower attitude, disavowing responsibility for the consequences of their work.

An epic data nerd battle started last Wednesday over Twitter as critics lambasted Public Policy Polling (PPP) for withholding a weekend poll of voters in Pueblo County, Colorado.

The poll asked Pueblo County voters if they supported the legislative recall of State Senator Angela Giron, a moderate Democrat who supported stronger gun control legislation in her state. Giron’s Tuesday recall election and that of State Senate President John Morse, of Colorado Springs, marked the first legislative recalls of state lawmakers in Colorado’s 137-year history.

PPP found 54 percent of Pueblo’s registered voters overall favored Giron’s recall, while just 42 percent opposed it. Stranger yet, a whopping 33 percent of Democratic respondents said that they supported the recall. Consider that 47 percent of Pueblo’s registered voters are Democrats; just 23 percent are Republicans.

“In a district that Barack Obama won by almost 20 points I figured there was no way that could be right,” a post from PPP Director Tom Jensen explained on the firm’s website. Not only that, voters in Pueblo County support expanded background checks for gun buyers—68 to 27 percent—and are split on ammunition limits.

“VERY bad and unscientific practice for @ppppolls to suppress a polling result they didn’t believe/didn’t like.”

PPP suspected a design flaw. “This was the first legislative recall election in Colorado history. There’s been a lot of voter confusion,” Jensen noted. “That finding made me think that respondents may not have understood what they were being asked.” With no time to test a new survey design, they waited for the election results.

Tuesday’s election results matched the unpublished poll results closely: 56 percent for the recall, 44 percent against. On Wednesday morning, PPP posted the weekend poll with an explanation for the delay and analysis. Poll trolling commenced.

PPP is not a public agency, it’s not a non-profit, and it’s not under any obligation to anyone but its clients to release its polls. The group has nevertheless made a policy of being responsive to public interest and shown a commitment to transparency. That’s laudable, not lamentable.

But Nate Silver couldn’t help himself: the much-feted statistician behind FiveThirtyEight threw a punch at the pollsters over Twitter early Wednesday afternoon, announcing, “VERY bad and unscientific practice for @ppppolls to suppress a polling result they didn’t believe/didn’t like.” He linked to the PPP post explaining their delay in releasing the Giron poll results. In his next-day follow-up, Silver accused PPP of an “approach to polling [that] is extremely ad hoc.” He added, “that ad-hockery stems from a lack of appreciation/understanding for the statistical fundamentals behind polling.”

Taking the cue from golden-boy Silver, other commentators and analysts have piled on. Mark Blumenthal, founding editor of and senior polling editor for the Huffington Post, rallied behind Silver by early afternoon Wednesday. He launched into his criticisms of PPP with a quick disclaimer: “I’m late to this #NerdFight.” Thursday brought a particularly pointed piece by Nate Cohn, The New Republic’s self-professed amateur polling analyst. Blumenthal rehashed Cohn’s arguments in a post for the Huffington Post.

It’s worth noting that, with the exception of Blumenthal, few of PPP’s critics have polling chops; journalists and practitioners with experience rallied behind PPP.

Silver criticizing a polling outfit for being reckless and unscientific is like Lindsay Lohan accusing Meryl Streep of lacking work ethic. In November 2011, Silver declared that President Obama had just a 17 percent chance of re-election. I wrote about the many problems with Silver’s piece, statistical and ethical, for the Huffington Post: Silver conceded he used just 17 elections (the post-1944 presidential cycles), discounted the reliability of historical precedent, admitted the inconsistency of economic forecasts, and exposed prevailing methods for predicting electoral outcome through tables and rundowns as no better than hit-or-miss.

In one corner, we have a hitherto respected polling outfit that releases data regularly; in the other, a statistician who strung together a series of untrustworthy variables to land the cover of the New York Times Magazine, justifying the exercise by listing the many problems with his claim in the body of his piece.

Gawker’s Max Read noted, “nate silver is a funny guy to be knocking an organization for lack of transparency!” The Plum Line’s Greg Sargent asked Nate Silver, “are you alleging they ‘suppressed’ the result because they ‘didn’t like’ it?” That “[PPP-Nate Silver] dispute turns out to be pretty narrow,” Brian Beutler of Salon commented. I found myself drawn into the Twitter war on the topic: “If you mixed blue and yellow and got purple, would you announce it before re-examining your experiment?”

It’s refreshing to find that there are still pollsters who self-regulate. Too many pollsters operating in the political sphere take an Ivory Tower attitude, disavowing responsibility for the consequences of their polls and analyses. Their reasoning, as I understand it, is that, so long as they note the margin of error and any other qualifications in fine print, they’re on the up-and-up. On that, they’re wrong.

Pre-election polling can affect election results—even Nate Silver admits as much. We’ve got multiple theories to choose from, each of which carries some weight. Voters may want to choose the winner or vote with their friends, for example. Low-information voters may take a marketplace approach, relying on public opinion to champion the worthiest candidate.

When a pollster puts out a poll with a huge margin of error or a skewed sample frame, most coverage won’t note the fine print, even if the pollster does. The irresponsible pollster publishes the poll, content to slap on a sensational headline or else criticize media for doing so. The responsible pollster recognizes the dangers and acts to ensure that data isn’t misappropriated.

PPP identified a problematic result and held it until they could explain what happened, despite the fact that they could’ve gotten publicity from the polling before the election. They published the poll after the election knowing they might face sustained criticism for having held it back. That’s a commitment to science and public responsibility that we should welcome.

Rebecca Buckwalter-Poza
Rebecca Buckwalter-Poza attended Harvard College and Yale Law School. She has written on law and politics for the Nation, the Atlantic, Politico, the Daily Beast, and CNN, and co-authored James Carville’s 40 More Years. Follow her on Twitter @rpbp.

More From Rebecca Buckwalter-Poza

A weekly roundup of the best of Pacific Standard and, delivered straight to your inbox.

Recent Posts

September 2 • 4:00 PM

Professors’ Pet Peeves

Ten things to avoid in your classrooms this year.

September 2 • 2:00 PM

Music Lessons Enhance Brain Function in Disadvantaged Kids

Children from poor neighborhoods in Los Angeles who took regular music lessons for two years were able to distinguish similar speech sounds faster than their peers.

September 2 • 12:00 PM

California Passes a Bill to Protect Workers in the Rapidly Growing Temp Staffing Industry

The bill will hold companies accountable for labor abuses by temp agencies and subcontractors they use.

September 2 • 10:00 AM

SWAT Pranks and SWAT Mistakes

The proliferation of risky police raids over the decades.

September 2 • 9:12 AM

Conference Call: The Graphic Novel

September 2 • 8:00 AM

Why We’re Not Holding State Legislators Accountable

The way we vote means that the political fortunes of state legislators hinge on events outside of their state and their control.

September 2 • 7:00 AM

When Men Who Abstain From Premarital Sex Get Married

Young men who take abstinence pledges have trouble adjusting to sexual norms when they become husbands.

September 2 • 6:00 AM

The Rise of Biblical Counseling

For millions of Christians, biblical counselors have replaced psychologists. Some think it’s time to reverse course.

September 2 • 5:12 AM

No Innovation Without Migration

People bring their ideas with them when they move from place to place.

September 2 • 4:00 AM

Why Middle School Doesn’t Have to Suck

Some people suspect the troubles of middle school are a matter of age. Middle schoolers, they think, are simply too moody, pimply, and cliquish to be easily educable. But these five studies might convince you otherwise.

September 2 • 3:13 AM

Coming Soon: When Robots Lie

September 2 • 2:00 AM

Introducing the New Issue of ‘Pacific Standard’

The science of self-control, the rise of biblical counseling, why middle school doesn’t have to suck, and more in our September/October 2014 print issue.

September 1 • 1:00 PM

Television and Overeating: What We Watch Matters

New research finds fast-moving programming leads to mindless overeating.

September 1 • 6:00 AM

Why Someone Named Monty Iceman Sold Doogie Howser’s Estate

How unusual names, under certain circumstances, can lead to success.

August 29 • 4:00 PM

The Hidden Costs of Tobacco Debt

Even when taxpayers aren’t explicitly on the hook, tobacco bonds can cost states and local governments money. Here’s how.

August 29 • 2:00 PM

Why Don’t Men and Women Wear the Same Gender-Neutral Bathing Suits?

They used to in the 1920s.

August 29 • 11:48 AM

Your Brain Decides Whether to Trust Someone in Milliseconds

We can determine trustworthiness even when we’re only subliminally aware of the other person.

August 29 • 10:00 AM

True Darwinism Is All About Chance

Though the rich sometimes forget, Darwin knew that nature frequently rolls the dice.

August 29 • 8:00 AM

Why Our Molecular Make-Up Can’t Explain Who We Are

Our genes only tell a portion of the story.

August 29 • 6:00 AM

Strange Situations: Attachment Theory and Sexual Assault on College Campuses

When college women leave home, does attachment behavior make them more vulnerable to campus rape?

August 29 • 4:00 AM

Forgive Your Philandering Partner—and Pay the Price

New research finds people who forgive an unfaithful romantic partner are considered weaker and less competent than those who ended the relationship.

August 28 • 4:00 PM

Some Natural-Looking Zoo Exhibits May Be Even Worse Than the Old Concrete Ones

They’re often designed for you, the paying visitor, and not the animals who have to inhabit them.

August 28 • 2:00 PM

What I Learned From Debating Science With Trolls

“Don’t feed the trolls” is sound advice, but occasionally ignoring it can lead to rewards.

Follow us

Subscribe Now

When Men Who Abstain From Premarital Sex Get Married

Young men who take abstinence pledges have trouble adjusting to sexual norms when they become husbands.

Your Brain Decides Whether to Trust Someone in Milliseconds

We can determine trustworthiness even when we’re only subliminally aware of the other person.

Young, Undocumented, and Invisible

While young migrant workers struggle under poor working conditions, U.S. policy has done little to help.

Education, Interrupted

When it comes to educational access, young Syrian refugees are becoming a “lost generation.”

No, Smartphone-Loss Anxiety Disorder Isn’t Real

But people are anxious about losing their phones, even if they don’t do much to protect them.

The Big One

One third of the United States federal budget for fighting wildfires goes toward one percent of such fires. September/October 2014 big-one-fires-final

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.