Menus Subscribe Search

Follow us


Quick Studies

Figure skater Fontana from Italy performs during the "2010 All That Skate Summer" ice show in Goyang

Figure skater Silvia Fontana from Italy. (Photo: queenyuna/Flickr)

Can You Learn to Judge Creativity?

• April 04, 2014 • 9:26 AM

Figure skater Silvia Fontana from Italy. (Photo: queenyuna/Flickr)

A new study suggests that, with training, amateurs can judge the level of creativity of artwork much like experts would. But is expert opinion always correct?

Once every four years, Americans gather around their television screens to partake in a national pastime, delivering brash, passionate opinions on a topic they barely understand—Olympic figure skating.

Of all the Olympic sports, figure skating is particularly attractive for the armchair judge because the scoring is more subjective than measuring a stick or declaring that a ball went into the net. Artistry ostensibly garners more points than a lifeless, mechanical performance.

Judging creativity, especially in high-stakes situations like the Olympics, is a contentious topic. Who is truly qualified to judge the creativity of a work of art or performance? And can you, as a casual, quadrennial ice-skating observer, ever rise to the ranks of a professional Olympic figure-skating judge?

A new research paper suggests that amateurs can, indeed, be trained to be better judges of creativity—at least when it comes to children’s paintings.

Study participants in the training group were given a brief lesson about the “subcomponents” of creativity, as determined by previous research. The training group participated in a practice round, where it rated the creativity level of paintings from one (not creative at all) to seven (very creative). Afterwards, the participants were told the “actual” creativity level, provided by a panel of expert judges.

“Non-trained judges seemed to agree on something else than the real creativity of the drawings.”

The control group, on the other hand, performed an unrelated task with the set of paintings. These subjects did not learn about the components of creativity.

The results showed that participants in the training group were more likely to excel at judging the creativity of children’s paintings in two distinct ways.

First, those who received training were more likely than the control group to deliver similar creativity ratings as the panel of expert judges.

Additionally, when the training group returned to the lab four weeks later to rate the same exact paintings, it tended to be more reliable, rating the paintings the same as it had in the first trial. The control group was not as reliable with its ratings.

Interestingly enough, while the control group participants’ ratings did not match the experts’, they did tend to agree with one another. In fact, they agreed with each other just as much as the experts and the training group. The authors write, “Non-trained judges seemed to agree on something else than the real creativity of the drawings.”

One mitigating factor, they hypothesize, could be that the group of participants was relatively homogenous—mostly young female university students.

The researchers say there’s no way to know what the untrained group was measuring. They emphasize that agreement with expert opinion should be the baseline of true creativity—after all, they do research creativity for a living.

Still, four years from now when you’re loudly debating the lack of spark in Ashley Wagner’s triple lutz—fifth beer of the night in hand—it’s good to know that while your opinion might not vibe with the experts, it’ll likely be the same as that of your barroom compatriots.

Bettina Chang
Associate Digital Editor Bettina Chang previously directed editorial content at HomeStyle and Real Estate Weekly. A Chicago native, she serves on the board of directors for Supplies for Dreams, working to improve education outcomes for Chicago Public Schools students. Follow her on Twitter @bechang8.

More From Bettina Chang

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts

December 17 • 4:00 PM

How to Run a Drug Dealing Network in Prison

People tend not to hear about the prison drug dealing operations that succeed. Substance.com asks a veteran of the game to explain his system.


December 17 • 2:00 PM

Gender Segregation of Toys Is on the Rise

Charting the use of “toys for boys” and “toys for girls” in American English.


December 17 • 12:41 PM

Why the College Football Playoff Is Terrible But Better Than Before

The sample size is still embarrassingly small, but at least there’s less room for the availability cascade.


December 17 • 11:06 AM

Canadian Kids Have a Serious Smoking Problem

Bootleg cigarette sales could be leading Canadian teens to more serious drugs, a recent study finds.


December 17 • 10:37 AM

A Public Lynching in Sproul Plaza

When photographs of lynching victims showed up on a hallowed site of democracy in action, a provocation was issued—but to whom, by whom, and why?


December 17 • 8:00 AM

What Was the Job?

This was the year the job broke, the year we accepted a re-interpretation of its fundamental bargain and bought in to the push to get us to all work for ourselves rather than each other.


December 17 • 6:00 AM

White Kids Will Be Kids

Even the “good” kids—bound for college, upwardly mobile—sometimes break the law. The difference? They don’t have much to fear. A professor of race and social movements reflects on her teenage years and faces some uncomfortable realities.



December 16 • 4:00 PM

How Fear of Occupy Wall Street Undermined the Red Cross’ Sandy Relief Effort

Red Cross responders say there was a ban on working with the widely praised Occupy Sandy relief group because it was seen as politically unpalatable.


December 16 • 3:30 PM

Murder! Mayhem! And That’s Just the Cartoons!

New research suggests deaths are common features of animated features aimed at children.


December 16 • 1:43 PM

In Tragedy, Empathy Still Dependent on Proximity

In spite of an increasingly connected world, in the face of adversity, a personal touch is most effective.


December 16 • 12:00 PM

The ‘New York Times’ Is Hooked on Drug du Jour Journalism

For the paper of record, addiction is always about this drug or that drug rather than the real causes.


December 16 • 10:00 AM

What Is the Point of Academic Books?

Ultimately, they’re meant to disseminate knowledge. But their narrow appeal makes them expensive to produce and harder to sell.


December 16 • 8:00 AM

Unjust and Unwell: The Racial Issues That Could Be Affecting Your Health Care

Physicians and medical students have the same problems with implicit bias as the rest of us.


December 16 • 6:00 AM

If You Get Confused Just Listen to the Music Play

Healing the brain with the Grateful Dead.


December 16 • 4:00 AM

Another Casualty of the Great Recession: Trust

Research from Britain finds people who were laid off from their jobs expressed lower levels of generalized trust.


December 15 • 4:00 PM

When Charter Schools Are Non-Profit in Name Only

Some charters pass along nearly all their money to for-profit companies hired to manage the schools. It’s an arrangement that’s raising eyebrows.


December 15 • 2:00 PM

No More Space Race

A far cry from the fierce Cold War Space Race between the U.S. and the Soviet Union, exploration in the 21st century is likely to be a much more globally collaborative project.


December 15 • 12:32 PM

The Hidden Psychology of the Home Ref

That old myth of home field bias isn’t a myth at all; it’s a statistical fact.


December 15 • 12:00 PM

Gluttony and Global Warming: We’re Eating Ourselves to a Warmer Planet

Forget your car. Our obsession with beef and dairy has a far more devastating effect on the climate.


December 15 • 10:00 AM

The 2016 Presidential Race Has Already Started

And this is the most exciting part.


December 15 • 8:00 AM

The Second Life of Old iPods

Why is it that old iPods are suddenly cool—and pricey again?


December 15 • 6:00 AM

The Lifelong Consequences of Rape

The long-term psychological and physical effects of the experience are devastating. And they’re likely exacerbated by the shame our culture insists on perpetuating.


December 15 • 4:00 AM

Mating Mindset Interferes With Attempts to Stop Smoking

Taiwanese researchers find photos of attractive women put men in an immediate-gratification state of mind.


December 15 • 2:00 AM

Where Innovation Thrives

Innovation does not require an urban area or a suburban area—it can happen in the city or in a small town. What it requires is open knowledge networks and the movement of people from different places.


Follow us


Canadian Kids Have a Serious Smoking Problem

Bootleg cigarette sales could be leading Canadian teens to more serious drugs, a recent study finds.

The Hidden Psychology of the Home Ref

That old myth of home field bias isn’t a myth at all; it’s a statistical fact.

A Word of Caution to the Holiday Deal-Makers

Repeat customers—with higher return rates and real bargain-hunting prowess—can have negative effects on a company’s net earnings.

Crowdfunding Works for Science

Scientists just need to put forth some effort.

There’s More Than One Way to Be Good at Math

Mathematical ability isn’t one single skill set; there are indeed many ways to be “good at math,” research shows.

The Big One

One in two United States senators and two in five House members who left office between 1998 and 2004 became lobbyists. November/December 2014

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.