Menus Subscribe Search

Follow us


Quick Studies

Figure skater Fontana from Italy performs during the "2010 All That Skate Summer" ice show in Goyang

Figure skater Silvia Fontana from Italy. (Photo: queenyuna/Flickr)

Can You Learn to Judge Creativity?

• April 04, 2014 • 9:26 AM

Figure skater Silvia Fontana from Italy. (Photo: queenyuna/Flickr)

A new study suggests that, with training, amateurs can judge the level of creativity of artwork much like experts would. But is expert opinion always correct?

Once every four years, Americans gather around their television screens to partake in a national pastime, delivering brash, passionate opinions on a topic they barely understand—Olympic figure skating.

Of all the Olympic sports, figure skating is particularly attractive for the armchair judge because the scoring is more subjective than measuring a stick or declaring that a ball went into the net. Artistry ostensibly garners more points than a lifeless, mechanical performance.

Judging creativity, especially in high-stakes situations like the Olympics, is a contentious topic. Who is truly qualified to judge the creativity of a work of art or performance? And can you, as a casual, quadrennial ice-skating observer, ever rise to the ranks of a professional Olympic figure-skating judge?

A new research paper suggests that amateurs can, indeed, be trained to be better judges of creativity—at least when it comes to children’s paintings.

Study participants in the training group were given a brief lesson about the “subcomponents” of creativity, as determined by previous research. The training group participated in a practice round, where it rated the creativity level of paintings from one (not creative at all) to seven (very creative). Afterwards, the participants were told the “actual” creativity level, provided by a panel of expert judges.

“Non-trained judges seemed to agree on something else than the real creativity of the drawings.”

The control group, on the other hand, performed an unrelated task with the set of paintings. These subjects did not learn about the components of creativity.

The results showed that participants in the training group were more likely to excel at judging the creativity of children’s paintings in two distinct ways.

First, those who received training were more likely than the control group to deliver similar creativity ratings as the panel of expert judges.

Additionally, when the training group returned to the lab four weeks later to rate the same exact paintings, it tended to be more reliable, rating the paintings the same as it had in the first trial. The control group was not as reliable with its ratings.

Interestingly enough, while the control group participants’ ratings did not match the experts’, they did tend to agree with one another. In fact, they agreed with each other just as much as the experts and the training group. The authors write, “Non-trained judges seemed to agree on something else than the real creativity of the drawings.”

One mitigating factor, they hypothesize, could be that the group of participants was relatively homogenous—mostly young female university students.

The researchers say there’s no way to know what the untrained group was measuring. They emphasize that agreement with expert opinion should be the baseline of true creativity—after all, they do research creativity for a living.

Still, four years from now when you’re loudly debating the lack of spark in Ashley Wagner’s triple lutz—fifth beer of the night in hand—it’s good to know that while your opinion might not vibe with the experts, it’ll likely be the same as that of your barroom compatriots.

Bettina Chang
Associate Digital Editor Bettina Chang previously directed editorial content at HomeStyle and Real Estate Weekly. A Chicago native, she serves on the board of directors for Supplies for Dreams, working to improve education outcomes for Chicago Public Schools students. Follow her on Twitter @bechang8.

More From Bettina Chang

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts

September 23 • 4:00 PM

A New Way Insurers Are Shifting Costs to the Sick

By charging higher prices for generic drugs that treat certain illness, health insurers may be violating the spirit of the Affordable Care Act, which bans discrimination against those with pre-existing conditions.


September 23 • 2:00 PM

Why Don’t More Women Commit Corporate Fraud?

Would having more female leaders reduce corporate crime? We don’t know, but the research suggests it’s likely.


September 23 • 12:00 PM

A Brief History of the Loch Ness Monster

From 1933—and possibly much, much earlier—to just this past May, people have been claiming (and staging) sightings of the famed water cryptid.



September 23 • 10:00 AM

The International Surrogacy Market

In Bangalore, where many women earn just $150 a month working in garment factories, surrogate mothers can make thousands of dollars by carrying others’ babies to term. But at what cost?


September 23 • 8:00 AM

Medicare: Your New Long-Term Care Provider

A 2013 court ruling has paved the way for an incredible, costly expansion of home health care by removing a critical lever the Centers for Medicare and Medicaid Services had to control who receives services, and for how long.


September 23 • 6:22 AM

On the Hunt for Fake Facebook Likes

A new study finds ways to uncover Facebook Like farms.


September 23 • 6:00 AM

The Heist: How Visitors Stole a National Monument

Fossil Cycad National Monument was home to one of the world’s greatest collections of fossilized cycadeoids—until visitors carried them all away.


September 23 • 4:00 AM

Fifty Shades of Meh

New research refutes the notion that reading the Fifty Shades of Grey trilogy strongly impacts women’s sexual behavior.


September 23 • 2:00 AM

The Portlandia Paradox

Oregon’s largest city is full of overeducated and underemployed young people.


September 22 • 4:00 PM

The Overly Harsh and Out-of-Date Law That’s So Difficult on Debtors

A 1968 federal law allows collectors to take 25 percent of debtors’ wages, or every penny in their bank accounts.


September 22 • 2:00 PM

NFL Players Are More Law Abiding Than Average Men

According to records kept by USA Today, 2.53 percent of players are arrested in any given year.


September 22 • 12:00 PM

Freaking Out About Outliers: When the Polls Are Way Off

The idea of such a small number of people being used to predict how millions will vote sometimes irks observers, but it’s actually a very reliable process—most of the time.


September 22 • 10:00 AM

The Imagined Sex Worker

The stigma against black sex workers can reinforce stigmas against all black women and all sex workers.


September 22 • 9:54 AM

All-Girls Schools Don’t Make Girls More Competitive

Parents, not educational setting, may be the key.


September 22 • 8:00 AM

The NFL, the Military, and the Problem With Masculine Institutions

Both the NFL and the U.S. military cultivate and reward a form of hyper-violent masculinity. The consequences of doing so have never been more obvious.


September 22 • 6:00 AM

Zombies in the Quad: The Trouble With Elite Education

William Deresiewicz’s new book, Excellent Sheep, is in part, he says, a letter to his younger, more privileged self.


September 22 • 4:02 AM

You’re Going to Die! So Buy Now!

New research finds inserting reminders of our mortality into advertisements is a surprisingly effective strategy to sell products.



September 19 • 4:00 PM

In Your Own Words: What It’s Like to Get Sued Over Past Debts

Some describe their surprise when they were sued after falling behind on medical and credit card bills.



September 19 • 1:26 PM

For Charitable Products, Sex Doesn’t Sell

Sexy women may turn heads, but for pro-social and charitable products, they won’t change minds.


September 19 • 12:00 PM

Carbon Taxes Really Do Work

A new study shows that taxing carbon dioxide emissions could actually work to reduce greenhouse gases without any negative effects on employment and revenues.


September 19 • 10:00 AM

Why the Poor Remain Poor

A follow-up to “How Being Poor Makes You Poor.”


September 19 • 9:03 AM

Why Science Won’t Defeat Ebola

While science will certainly help, winning the battle against Ebola is a social challenge.


Follow us


On the Hunt for Fake Facebook Likes

A new study finds ways to uncover Facebook Like farms.

All-Girls Schools Don’t Make Girls More Competitive

Parents, not educational setting, may be the key.

For Charitable Products, Sex Doesn’t Sell

Sexy women may turn heads, but for pro-social and charitable products, they won't change minds.

Carbon Taxes Really Do Work

A new study shows that taxing carbon dioxide emissions could actually work to reduce greenhouse gases without any negative effects on employment and revenues.

Savor Good Times, Get Through the Bad Ones—With Categories

Ticking off a category of things to do can feel like progress or a fun time coming to an end.

The Big One

One in three tourists to Jamaica reports getting harassed; half of them are hassled to buy drugs. September/October 2014 new-big-one-4

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.