Menus Subscribe Search

Follow us


Quick Studies

Figure skater Fontana from Italy performs during the "2010 All That Skate Summer" ice show in Goyang

Figure skater Silvia Fontana from Italy. (Photo: queenyuna/Flickr)

Can You Learn to Judge Creativity?

• April 04, 2014 • 9:26 AM

Figure skater Silvia Fontana from Italy. (Photo: queenyuna/Flickr)

A new study suggests that, with training, amateurs can judge the level of creativity of artwork much like experts would. But is expert opinion always correct?

Once every four years, Americans gather around their television screens to partake in a national pastime, delivering brash, passionate opinions on a topic they barely understand—Olympic figure skating.

Of all the Olympic sports, figure skating is particularly attractive for the armchair judge because the scoring is more subjective than measuring a stick or declaring that a ball went into the net. Artistry ostensibly garners more points than a lifeless, mechanical performance.

Judging creativity, especially in high-stakes situations like the Olympics, is a contentious topic. Who is truly qualified to judge the creativity of a work of art or performance? And can you, as a casual, quadrennial ice-skating observer, ever rise to the ranks of a professional Olympic figure-skating judge?

A new research paper suggests that amateurs can, indeed, be trained to be better judges of creativity—at least when it comes to children’s paintings.

Study participants in the training group were given a brief lesson about the “subcomponents” of creativity, as determined by previous research. The training group participated in a practice round, where it rated the creativity level of paintings from one (not creative at all) to seven (very creative). Afterwards, the participants were told the “actual” creativity level, provided by a panel of expert judges.

“Non-trained judges seemed to agree on something else than the real creativity of the drawings.”

The control group, on the other hand, performed an unrelated task with the set of paintings. These subjects did not learn about the components of creativity.

The results showed that participants in the training group were more likely to excel at judging the creativity of children’s paintings in two distinct ways.

First, those who received training were more likely than the control group to deliver similar creativity ratings as the panel of expert judges.

Additionally, when the training group returned to the lab four weeks later to rate the same exact paintings, it tended to be more reliable, rating the paintings the same as it had in the first trial. The control group was not as reliable with its ratings.

Interestingly enough, while the control group participants’ ratings did not match the experts’, they did tend to agree with one another. In fact, they agreed with each other just as much as the experts and the training group. The authors write, “Non-trained judges seemed to agree on something else than the real creativity of the drawings.”

One mitigating factor, they hypothesize, could be that the group of participants was relatively homogenous—mostly young female university students.

The researchers say there’s no way to know what the untrained group was measuring. They emphasize that agreement with expert opinion should be the baseline of true creativity—after all, they do research creativity for a living.

Still, four years from now when you’re loudly debating the lack of spark in Ashley Wagner’s triple lutz—fifth beer of the night in hand—it’s good to know that while your opinion might not vibe with the experts, it’ll likely be the same as that of your barroom compatriots.

Bettina Chang
Associate Digital Editor Bettina Chang previously directed editorial content at HomeStyle and Real Estate Weekly. A Chicago native, she serves on the board of directors for Supplies for Dreams, working to improve education outcomes for Chicago Public Schools students. Follow her on Twitter @bechang8.

More From Bettina Chang

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts


September 30 • 10:09 AM

Trust Is Waning, and Inequality May Be to Blame

Trust in others and confidence in institutions is declining, while economic inequality creeps up, a new study shows.


September 30 • 8:00 AM

The Psychology of Penmanship

Graphology: It’s all (probably) bunk.



September 30 • 6:00 AM

The Medium Is the Message, 50 Years Later

Five decades on, what can Marshall McLuhan’s Understanding Media tell us about today?


September 30 • 4:00 AM

Grad School’s Mental Health Problem

Navigating the emotional stress of doctoral programs in a down market.


September 29 • 1:21 PM

Conference Call: Free Will Conference


September 29 • 12:00 PM

How Copyright Law Protects Art From Criticism

A case for allowing the copyright on Gone With the Wind to expire.


September 29 • 10:00 AM

Should We Be Told Who Funds Political Attack Ads?

On the value of campaign finance disclosure.


September 29 • 8:00 AM

Searching for a Man Named Penis

A quest to track down a real Penis proves difficult.


September 29 • 6:00 AM

Why Do So Many People Watch HGTV?

The same reason so many people watch NCIS or Law and Order: It’s all a procedural.


September 29 • 4:00 AM

The Link Between Depression and Terrorism

A new study from the United Kingdom finds a connection between depression and radicalization.


September 26 • 4:00 PM

Fast Track to a Spill?

Oil pipeline projects across America are speeding forward without environmental review.


September 26 • 2:00 PM

Why Liberals Love the Disease Theory of Addiction, by a Liberal Who Hates It

The disease model is convenient to liberals because it spares them having to say negative things about poor communities. But this conception of addiction harms the very people we wish to help.


September 26 • 1:21 PM

Race, Trust, and Split-Second Judgments


September 26 • 9:47 AM

Dopamine Might Be Behind Impulsive Behavior

A monkey study suggests the brain chemical makes what’s new and different more attractive.


September 26 • 8:00 AM

A Letter Becomes a Book Becomes a Play

Sarah Ruhl’s Dear Elizabeth: A Play in Letters From Elizabeth Bishop to Robert Lowell and Back Again takes 900 pages of correspondence between the two poets and turns them into an on-stage performance.


September 26 • 7:00 AM

Sonic Hedgehog, DICER, and the Problem With Naming Genes

Wait, why is there a Pokemon gene?


September 26 • 6:00 AM

Sounds Like the Blues

At a music-licensing firm, any situation can become nostalgic, romantic, or adventurous, given the right background sounds.


September 26 • 5:00 AM

The Dark Side of Empathy

New research finds the much-lauded feeling of identification with another person’s emotions can lead to unwarranted aggressive behavior.



September 25 • 4:00 PM

Forging a New Path: Working to Build the Perfect Wildlife Corridor

When it comes to designing wildlife corridors, our most brilliant analytical minds are still no match for Mother Nature. But we’re getting there.


September 25 • 2:00 PM

Fashion as a Inescapable Institution

Like it or not, fashion is an institution because we can no longer feasibly make our own clothes.


September 25 • 12:00 PM

The Fake Birth Mothers Who Bilk Couples Out of Their Cash by Promising Future Babies

Another group that’s especially vulnerable to scams and fraud is that made up of those who are desperate to adopt a child.


September 25 • 10:03 AM

The Way We QuickType


Follow us


Trust Is Waning, and Inequality May Be to Blame

Trust in others and confidence in institutions is declining, while economic inequality creeps up, a new study shows.

Dopamine Might Be Behind Impulsive Behavior

A monkey study suggests the brain chemical makes what's new and different more attractive.

School Counselors Do More Than You’d Think

Adding just one counselor to a school has an enormous impact on discipline and test scores, according to a new study.

How a Second Language Trains Your Brain for Math

Second languages strengthen the brain's executive control circuits, with benefits beyond words.

Would You Rather Go Blind or Lose Your Mind?

Americans consistently fear blindness, but how they compare it to other ailments varies across racial lines.

The Big One

One company, Amazon, controls 67 percent of the e-book market in the United States—down from 90 percent five years ago. September/October 2014 new-big-one-5

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.