Menus Subscribe Search
digital-polling

(ILLUSTRATION: PAVEL IGNATOV/SHUTTERSTOCK)

The Use and Abuse of Polling in American Politics

• September 16, 2013 • 8:00 AM

(ILLUSTRATION: PAVEL IGNATOV/SHUTTERSTOCK)

Last week’s fight over PPP’s decision to hold back the results of a poll highlight how too many pollsters operating in the political sphere take an Ivory Tower attitude, disavowing responsibility for the consequences of their work.

An epic data nerd battle started last Wednesday over Twitter as critics lambasted Public Policy Polling (PPP) for withholding a weekend poll of voters in Pueblo County, Colorado.

The poll asked Pueblo County voters if they supported the legislative recall of State Senator Angela Giron, a moderate Democrat who supported stronger gun control legislation in her state. Giron’s Tuesday recall election and that of State Senate President John Morse, of Colorado Springs, marked the first legislative recalls of state lawmakers in Colorado’s 137-year history.

PPP found 54 percent of Pueblo’s registered voters overall favored Giron’s recall, while just 42 percent opposed it. Stranger yet, a whopping 33 percent of Democratic respondents said that they supported the recall. Consider that 47 percent of Pueblo’s registered voters are Democrats; just 23 percent are Republicans.

“In a district that Barack Obama won by almost 20 points I figured there was no way that could be right,” a post from PPP Director Tom Jensen explained on the firm’s website. Not only that, voters in Pueblo County support expanded background checks for gun buyers—68 to 27 percent—and are split on ammunition limits.

“VERY bad and unscientific practice for @ppppolls to suppress a polling result they didn’t believe/didn’t like.”

PPP suspected a design flaw. “This was the first legislative recall election in Colorado history. There’s been a lot of voter confusion,” Jensen noted. “That finding made me think that respondents may not have understood what they were being asked.” With no time to test a new survey design, they waited for the election results.

Tuesday’s election results matched the unpublished poll results closely: 56 percent for the recall, 44 percent against. On Wednesday morning, PPP posted the weekend poll with an explanation for the delay and analysis. Poll trolling commenced.

PPP is not a public agency, it’s not a non-profit, and it’s not under any obligation to anyone but its clients to release its polls. The group has nevertheless made a policy of being responsive to public interest and shown a commitment to transparency. That’s laudable, not lamentable.

But Nate Silver couldn’t help himself: the much-feted statistician behind FiveThirtyEight threw a punch at the pollsters over Twitter early Wednesday afternoon, announcing, “VERY bad and unscientific practice for @ppppolls to suppress a polling result they didn’t believe/didn’t like.” He linked to the PPP post explaining their delay in releasing the Giron poll results. In his next-day follow-up, Silver accused PPP of an “approach to polling [that] is extremely ad hoc.” He added, “that ad-hockery stems from a lack of appreciation/understanding for the statistical fundamentals behind polling.”

Taking the cue from golden-boy Silver, other commentators and analysts have piled on. Mark Blumenthal, founding editor of Pollster.com and senior polling editor for the Huffington Post, rallied behind Silver by early afternoon Wednesday. He launched into his criticisms of PPP with a quick disclaimer: “I’m late to this #NerdFight.” Thursday brought a particularly pointed piece by Nate Cohn, The New Republic’s self-professed amateur polling analyst. Blumenthal rehashed Cohn’s arguments in a post for the Huffington Post.

It’s worth noting that, with the exception of Blumenthal, few of PPP’s critics have polling chops; journalists and practitioners with experience rallied behind PPP.

Silver criticizing a polling outfit for being reckless and unscientific is like Lindsay Lohan accusing Meryl Streep of lacking work ethic. In November 2011, Silver declared that President Obama had just a 17 percent chance of re-election. I wrote about the many problems with Silver’s piece, statistical and ethical, for the Huffington Post: Silver conceded he used just 17 elections (the post-1944 presidential cycles), discounted the reliability of historical precedent, admitted the inconsistency of economic forecasts, and exposed prevailing methods for predicting electoral outcome through tables and rundowns as no better than hit-or-miss.

In one corner, we have a hitherto respected polling outfit that releases data regularly; in the other, a statistician who strung together a series of untrustworthy variables to land the cover of the New York Times Magazine, justifying the exercise by listing the many problems with his claim in the body of his piece.

Gawker’s Max Read noted, “nate silver is a funny guy to be knocking an organization for lack of transparency!” The Plum Line’s Greg Sargent asked Nate Silver, “are you alleging they ‘suppressed’ the result because they ‘didn’t like’ it?” That “[PPP-Nate Silver] dispute turns out to be pretty narrow,” Brian Beutler of Salon commented. I found myself drawn into the Twitter war on the topic: “If you mixed blue and yellow and got purple, would you announce it before re-examining your experiment?”

It’s refreshing to find that there are still pollsters who self-regulate. Too many pollsters operating in the political sphere take an Ivory Tower attitude, disavowing responsibility for the consequences of their polls and analyses. Their reasoning, as I understand it, is that, so long as they note the margin of error and any other qualifications in fine print, they’re on the up-and-up. On that, they’re wrong.

Pre-election polling can affect election results—even Nate Silver admits as much. We’ve got multiple theories to choose from, each of which carries some weight. Voters may want to choose the winner or vote with their friends, for example. Low-information voters may take a marketplace approach, relying on public opinion to champion the worthiest candidate.

When a pollster puts out a poll with a huge margin of error or a skewed sample frame, most coverage won’t note the fine print, even if the pollster does. The irresponsible pollster publishes the poll, content to slap on a sensational headline or else criticize media for doing so. The responsible pollster recognizes the dangers and acts to ensure that data isn’t misappropriated.

PPP identified a problematic result and held it until they could explain what happened, despite the fact that they could’ve gotten publicity from the polling before the election. They published the poll after the election knowing they might face sustained criticism for having held it back. That’s a commitment to science and public responsibility that we should welcome.

Rebecca Buckwalter-Poza
Rebecca Buckwalter-Poza attended Harvard College and Yale Law School. She has written on law and politics for the Nation, the Atlantic, Politico, the Daily Beast, and CNN, and co-authored James Carville’s 40 More Years. Follow her on Twitter @rpbp.

More From Rebecca Buckwalter-Poza

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts

September 22 • 2:00 PM

NFL Players Are More Law Abiding Than Average Men

According to records kept by USA Today, 2.53 percent of players are arrested in any given year.


September 22 • 12:00 PM

Freaking Out About Outliers: When the Polls Are Way Off

The idea of such a small number of people being used to predict how millions will vote sometimes irks observers, but it’s actually a very reliable process—most of the time.


September 22 • 10:00 AM

The Imagined Sex Worker

The stigma against black sex workers can reinforce stigmas against all black women and all sex workers.


September 22 • 9:54 AM

All-Girls Schools Don’t Make Girls More Competitive

Parents, not educational setting, may be the key.


September 22 • 8:00 AM

The NFL, the Military, and the Problem With Masculine Institutions

Both the NFL and the U.S. military cultivate and reward a form of hyper-violent masculinity. The consequences of doing so have never been more obvious.


September 22 • 6:00 AM

Zombies in the Quad: The Trouble With Elite Education

William Deresiewicz’s new book, Excellent Sheep, is in part, he says, a letter to his younger, more privileged self.


September 22 • 4:02 AM

You’re Going to Die! So Buy Now!

New research finds inserting reminders of our mortality into advertisements is a surprisingly effective strategy to sell products.



September 19 • 4:00 PM

In Your Own Words: What It’s Like to Get Sued Over Past Debts

Some describe their surprise when they were sued after falling behind on medical and credit card bills.



September 19 • 1:26 PM

For Charitable Products, Sex Doesn’t Sell

Sexy women may turn heads, but for pro-social and charitable products, they won’t change minds.


September 19 • 12:00 PM

Carbon Taxes Really Do Work

A new study shows that taxing carbon dioxide emissions could actually work to reduce greenhouse gases without any negative effects on employment and revenues.


September 19 • 10:00 AM

Why the Poor Remain Poor

A follow-up to “How Being Poor Makes You Poor.”


September 19 • 9:03 AM

Why Science Won’t Defeat Ebola

While science will certainly help, winning the battle against Ebola is a social challenge.


September 19 • 8:00 AM

Burrito Treason in the Lone Star State

Did Meatless Mondays bring down Texas Agriculture Commissioner Todd Staples?


September 19 • 7:31 AM

Savor Good Times, Get Through the Bad Ones—With Categories

Ticking off a category of things to do can feel like progress or a fun time coming to an end.


September 19 • 6:00 AM

The Most Untouchable Man in Sports

How the head of the governing body for the world’s most popular sport freely wields his wildly incompetent power.


September 19 • 4:00 AM

The Danger of Dining With an Overweight Companion

There’s a good chance you’ll eat more unhealthy food.



September 18 • 4:00 PM

Racial Disparity in Imprisonment Inspires White People to Be Even More Tough on Crime

White Americans are more comfortable with punitive and harsh policing and sentencing when they imagine that the people being policed and put in prison are black.



September 18 • 2:00 PM

The Wages of Millions Are Being Seized to Pay Past Debts

A new study provides the first-ever tally of how many employees lose up to a quarter of their paychecks over debts like unpaid credit card or medical bills and student loans.


September 18 • 12:00 PM

When Counterfeit and Contaminated Drugs Are Deadly

The cost and the crackdown, worldwide.


September 18 • 10:00 AM

How Do You Make a Living, Molly Crabapple?

Noah Davis talks to Molly Crapabble about Michelangelo, the Medicis, and the tension between making art and making money.


September 18 • 9:00 AM

Um, Why Are These Professors Creeping on My Facebook Page?

The ethics of student-teacher “intimacy”—on campus and on social media.


Follow us


All-Girls Schools Don’t Make Girls More Competitive

Parents, not educational setting, may be the key.

For Charitable Products, Sex Doesn’t Sell

Sexy women may turn heads, but for pro-social and charitable products, they won't change minds.

Carbon Taxes Really Do Work

A new study shows that taxing carbon dioxide emissions could actually work to reduce greenhouse gases without any negative effects on employment and revenues.

Savor Good Times, Get Through the Bad Ones—With Categories

Ticking off a category of things to do can feel like progress or a fun time coming to an end.

How to Build a Better Election

Elimination-style voting is harder to fiddle with than majority rule.

The Big One

One in three tourists to Jamaica reports getting harassed; half of them are hassled to buy drugs. September/October 2014 new-big-one-4

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.