Menus Subscribe Search

The World Wide Web

secret-service

(Photo: David Stuart Productions/Shutterstock)

Can We Really Detect Sarcasm With a Machine?

• June 17, 2014 • 10:00 AM

(Photo: David Stuart Productions/Shutterstock)

What was once the domain of literary critics has now become the world of the Secret Service.

Eyes rolled a couple of weeks ago when the Secret Service posted a work order for new social media analytics software. Not because the law enforcement agency tasked with protecting the president and other federal officials lacks a social media presence, but because among the 22 functionality requirements for the software was the “ability to detect sarcasm and false positives.”

Edwin M. Donovan, deputy assistant director of the Secret Service’s Office of Government and Public Affairs, emphasizes that sarcasm detection is only one of many requirements. “It’s so you don’t have to sift through thousands of tweets,” he says. “We want to streamline, to automate the social media monitoring process. Not just for sarcasm.”

“Right now, if there were suddenly 200 tweets about something on D Street near the Capitol, we’d want to be able to synthesize that.”

Donovan says that assessing large volumes of messages has been a problem for the Secret Service, which in the past has borrowed the analytics tools of other agencies, like FEMA. “It’s an issue that’s come up before, and we don’t want to waste time sifting through messages.”

“Right now,” Donovan says, “if there were suddenly 200 tweets about something on D Street near the Capitol, we’d want to be able to synthesize that.” The agency wants to be able to monitor its own presence on social media, but also track trending topics and influential users.

“Remember the purple tunnel of doom?” Donovan asks, referring to the debacle during the 2009 presidential inauguration when thousands of attendances with purple tickets were caught in the Third Street Tunnel, unable to cross 395 to attend the event. “We weren’t monitoring Twitter that day, so we didn’t know. Since then, we’ve entered social media.”

The Secret Service (@SecretService) joined Twitter in February of 2010. The account has posted only 572 tweets, but is followed by 110,000 users. And it’s not the only governmental agency interested in assessing the tone of other users. Last summer, the BBC reported that the French company Spotter provided such an analytics tool to the British Home Office, the European Union Commission, and Dubai Courts.

For around $1,675 per month, Spotter provides software that “uses a combination of linguistics, semantics and heuristics to create algorithms that generate reports about online reputation.” Determining whether users are genuinely complaining or sincerely threatening a government agency is difficult, but Spotter claims an 80 percent accuracy rate and offers the service in 29 languages.

But how does software accomplish a task that is difficult even for sentient beings? Even the most casual of correspondents has had a text message or email misunderstood by its recipient. Without the aid of eye rolls or shoulder shrugs, textual sarcasm can be difficult to detect. The science of sincerity, or sarcasm if you’re a glass-empty type, is critical for law enforcement agencies, but also profitable for corporations seeking to improve their customer service. Understanding whether a reviewer sarcastically praised your product or honestly questioned your services is critical.

So what used to be the domain of literary critics has become a novel problem for software designers. A 2010 paper by three researchers at Hebrew University presented a sarcasm algorithm with a 77 percent precision rate. Developed through an analysis of 66,000 Amazon product reviews, the algorithm was able to detect the situational irony of titles (“[I] Love the Cover” for a book review and “Where Am I?” for a GPS device) and sarcastic patterns in speech and punctuation (sentence length, multiple exclamation or question marks in sentences, and the number of words with all capital letters among others).

But Amazon reviews have many words. In a 2011 paper, three researchers at Rutgers’ School of Communication & Information were much less successful at identifying sarcasm on Twitter, which is limited to 140-character tweets. Tweet length as well as the lack of context made assessing tone very difficult, for machines but also humans. Even with the assistance of Twitter tics like emoticons and hashtags, the human judges and the machines demonstrated only 70 percent accuracy when distinguishing sarcastic tweets from positives or negative tweets.

But that was 2011, and with companies like Spotter already reporting 80 percent accuracy, the Secret Service will likely find more bids than it needs. Donovan expressed some sarcasm of his own when I asked about how they will measure the actual accuracy of whichever software they purchase. “That’s a great question,” Donovan says, “for the companies that make these claims.”

Casey N. Cep
Casey N. Cep is a writer from the Eastern Shore of Maryland. She has written for the New Republic, the New York Times, the New Yorker, and the Paris Review. Follow her on Twitter @cncep.

More From Casey N. Cep

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts

September 16 • 12:00 PM

What Makes You So Smart, Brilliant 12-Year-Old?

Charles Wang is going to rule the world.


September 16 • 10:09 AM

No Innovation Without Migration: The Harlem Renaissance

The Harlem Renaissance wasn’t a place, but an era of migration. It would have happened even without New York City.


September 16 • 10:00 AM

A Law Professor Walks Into a Creative Writing Workshop

One academic makes the case for learning how to write.



September 16 • 7:23 AM

Does Not Checking Your Buddy’s Facebook Updates Make You a Bad Friend?

An etiquette expert, a social scientist, and an old pal of mine ponder the ever-shifting rules of friendship.



September 16 • 6:12 AM

3-D Movies Aren’t That Special

Psychologists find that 3-D doesn’t have any extra emotional impact.


September 16 • 6:00 AM

What Color Is Your Pygmy Goat?

The fierce battle over genetic purity, writ small. Very small.



September 15 • 4:00 PM

The Average Prisoner Is Visited Only Twice While Incarcerated

And black prisoners receive even fewer visitors.


September 15 • 2:00 PM

Gambling With America’s Health

The public health costs of legal gambling.


September 15 • 12:23 PM

The Scent of a Conservative

We are attracted to the body odor of others with similar political beliefs, according to new research.


September 15 • 12:00 PM

2014: A Pretty Average Election

Don’t get too worked up over this year’s congressional mid-terms.


September 15 • 10:00 AM

Online Harassment of Women Isn’t Just a Gamer Problem

By blaming specific subcultures, we ignore a much larger and more troubling social pathology.


September 15 • 8:00 AM

Atheists Seen as a Threat to Moral Values

New research attempts to pinpoint why non-believers are widely disliked and distrusted.


September 15 • 6:12 AM

To Protect Against Meltdowns, Banks Must Map Financial Interconnections

A new model suggests looking beyond balance sheets, studying the network of investment as well.


September 15 • 6:00 AM

Interview With a Drug Dealer

What happens when the illicit product you’ve made your living off of finally becomes legal?


September 15 • 4:00 AM

A Feeling of Control: How America Can Finally Learn to Deal With Its Impulses

The ability to delay gratification has been held up as the one character trait to rule them all—the key to academic success, financial security, and social well-being. But willpower isn’t the answer. The new, emotional science of self-regulation.



September 15 • 2:04 AM

No Innovation Without Migration: Do Places Make People?

We know that people make places, but does it also work the other way?


September 12 • 4:00 PM

Life, Liberty, and the Pursuit of Plastic Bags

California wants you to pay for your plastic bags. (FYI: That’s not an infringement on your constitutional rights.)


September 12 • 2:00 PM

Should We Trust the Hearts of White People?

On the 50th anniversary of the Civil Rights Act, revisiting a clip of James Baldwin on the Dick Cavett Show.


September 12 • 12:00 PM

Big Government, Happy Citizens?

You may like to talk about how much happier you’d be if the government didn’t interfere with your life, but that’s not what the research shows.


September 12 • 10:00 AM

Whispering in the Town Square: Can Twitter Provide an Escape From All Its Noise?

Twitter has created its own buzzing, digital agora, but when users want to speak amongst themselves, they tend to leave for another platform. It’s a social network that helps you find people to talk to—but barely lets you do any talking.


September 12 • 9:03 AM

How Ancient DNA Is Rewriting Human History

We thought we knew how we’d been shaped by evolution. We were wrong.


Follow us


3-D Movies Aren’t That Special

Psychologists find that 3-D doesn't have any extra emotional impact.

To Protect Against Meltdowns, Banks Must Map Financial Interconnections

A new model suggests looking beyond balance sheets, studying the network of investment as well.

Big Government, Happy Citizens?

You may like to talk about how much happier you'd be if the government didn't interfere with your life, but that's not what the research shows.

Give Yourself a Present for the Future

Psychologists discover that we underestimate the value of looking back.

In Soccer as in Art, Motifs Matter

A new study suggests a way to quantitatively measure a team’s style through its pass flow. It may become another metric used to evaluate potential recruits.

The Big One

One in three drivers in Brooklyn's Park Slope—at certain times of day—is just looking for parking. The same goes for drivers in Manhattan's SoHo. September/October 2014 new-big-one-3

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.