Menus Subscribe Search

Quick Studies

Figure skater Fontana from Italy performs during the "2010 All That Skate Summer" ice show in Goyang

Figure skater Silvia Fontana from Italy. (Photo: queenyuna/Flickr)

Can You Learn to Judge Creativity?

• April 04, 2014 • 9:26 AM

Figure skater Silvia Fontana from Italy. (Photo: queenyuna/Flickr)

A new study suggests that, with training, amateurs can judge the level of creativity of artwork much like experts would. But is expert opinion always correct?

Once every four years, Americans gather around their television screens to partake in a national pastime, delivering brash, passionate opinions on a topic they barely understand—Olympic figure skating.

Of all the Olympic sports, figure skating is particularly attractive for the armchair judge because the scoring is more subjective than measuring a stick or declaring that a ball went into the net. Artistry ostensibly garners more points than a lifeless, mechanical performance.

Judging creativity, especially in high-stakes situations like the Olympics, is a contentious topic. Who is truly qualified to judge the creativity of a work of art or performance? And can you, as a casual, quadrennial ice-skating observer, ever rise to the ranks of a professional Olympic figure-skating judge?

A new research paper suggests that amateurs can, indeed, be trained to be better judges of creativity—at least when it comes to children’s paintings.

Study participants in the training group were given a brief lesson about the “subcomponents” of creativity, as determined by previous research. The training group participated in a practice round, where it rated the creativity level of paintings from one (not creative at all) to seven (very creative). Afterwards, the participants were told the “actual” creativity level, provided by a panel of expert judges.

“Non-trained judges seemed to agree on something else than the real creativity of the drawings.”

The control group, on the other hand, performed an unrelated task with the set of paintings. These subjects did not learn about the components of creativity.

The results showed that participants in the training group were more likely to excel at judging the creativity of children’s paintings in two distinct ways.

First, those who received training were more likely than the control group to deliver similar creativity ratings as the panel of expert judges.

Additionally, when the training group returned to the lab four weeks later to rate the same exact paintings, it tended to be more reliable, rating the paintings the same as it had in the first trial. The control group was not as reliable with its ratings.

Interestingly enough, while the control group participants’ ratings did not match the experts’, they did tend to agree with one another. In fact, they agreed with each other just as much as the experts and the training group. The authors write, “Non-trained judges seemed to agree on something else than the real creativity of the drawings.”

One mitigating factor, they hypothesize, could be that the group of participants was relatively homogenous—mostly young female university students.

The researchers say there’s no way to know what the untrained group was measuring. They emphasize that agreement with expert opinion should be the baseline of true creativity—after all, they do research creativity for a living.

Still, four years from now when you’re loudly debating the lack of spark in Ashley Wagner’s triple lutz—fifth beer of the night in hand—it’s good to know that while your opinion might not vibe with the experts, it’ll likely be the same as that of your barroom compatriots.

Bettina Chang
Associate Digital Editor Bettina Chang previously directed editorial content at HomeStyle and Real Estate Weekly. A Chicago native, she serves on the board of directors for Supplies for Dreams, working to improve education outcomes for Chicago Public Schools students. Follow her on Twitter @bechang8.

More From Bettina Chang

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts

August 20 • 8:40 AM

Psychopathic or Just Antisocial? A Key Brain Difference Tells the Tale

Though psychopaths and antisocial people may seem similar, what occurs in their brains isn’t.


August 20 • 8:00 AM

What the Cost of Raising a Child in America Tells Us About Income Inequality

You’ll spend nearly a quarter of a million dollars to raise a kid in the United States, or about five times the annual median income.


August 20 • 6:00 AM

In Praise of ‘American Greed’

While it remains semi-hidden on CNBC and can’t claim the car chases of Cops, American Greed—now with eight seasons in the books—has proven itself a worthy endeavor.


August 20 • 4:00 AM

Of Course I Behaved Like a Jerk, I Was Just Watching ‘Jersey Shore’

Researchers find watching certain types of reality TV can make viewers more aggressive.


August 20 • 2:00 AM

Concluding Remarks About Housing Affordability and Supply Restricitions

Demand, not supply, plays the dominant role in explaining the housing affordability crisis. The wages are just too damn low.


August 19 • 4:00 PM

Can Lawmakers Only Make Laws That Corporations Allow?

There’s a telling detail in a recent story about efforts to close loopholes in corporate tax laws.




August 19 • 12:00 PM

How ‘Contagion’ Became Contagious

Do ideas and emotions really spread like a virus?


August 19 • 10:00 AM

Child Refugees: The New Barbarians

The disturbing rhetoric around the recent rise in child refugees into the United States from Central America may be shaping popular opinion on upcoming immigration reform.


August 19 • 8:00 AM

Making Police Departments More Diverse Isn’t Enough

Local police departments should reflect the communities they serve, but fixing that alone won’t curb unnecessary violence.


August 19 • 7:15 AM

Common Knowledge Makes Us More Cooperative

People are more inclined to take mutually beneficial risks if they know what others know.


August 19 • 6:00 AM

Seeking a Healthy Public School Lunch? Good Luck

Mystery meat will always win.


August 19 • 4:00 AM

The Positive Effects of Sports-Themed Video Games

New research finds sports-themed video games actually encourage some kids to get onto the field.


August 19 • 1:00 AM

DIY Diagnosis: How an Extreme Athlete Uncovered Her Genetic Flaw

When Kim Goodsell discovered that she had two extremely rare genetic diseases, she taught herself genetics to help find out why.



August 18 • 3:30 PM

Mister Rogers’ Heart-Healthy Neighborhood

Researchers find living in a friendly, cohesive neighborhood lowers seniors’ chances of having a heart attack.


August 18 • 2:00 PM

Wealth or Good Parenting?

Framing the privileges of the rich.


August 18 • 12:00 PM

How Much Did the Stigma of Mental Illness Harm Robin Williams?

Addiction treatment routinely fails people with mental illnesses, while mental health care often ignores addiction. And everywhere, stigma is rife. Can a tragic death prompt a more intelligent approach?


August 18 • 10:00 AM

Punished for Being Poor: The Problem With Using Big Data in the Justice System

Correctional departments use data-driven analyses because they’re easier and cheaper than individual assessments. But at what cost?


August 18 • 8:00 AM

What Americans Can Learn From a Vial of Tibetan Spit

Living high in the mountains for thousands of years, Tibetans have developed distinct biological traits that could benefit all of us, but translating medical science across cultures is always a tricky business.


August 18 • 6:00 AM

The Problems With William Deresiewicz’s New Manifesto

Excellent Sheep: a facile approach to an urgent critique.


August 18 • 4:00 AM

Ferguson Is a Serious Outlier

One black city council member is not nearly enough. In a study of city councils, only one place in America had a greater representational disparity than Ferguson, Missouri.


August 16 • 4:00 AM

Six Days in Ferguson: Voices From the Protests

A day-by-day chronology of what happened in Ferguson, drawn from the best reporting by journalists and witnesses on the ground.


August 15 • 4:00 PM

Skirting Ochobo: Big Business Finds a Way Around Local Customs

The “liberation wrapper,” which was designed to shield mouths from public view while eating, has helped a Japanese chain increase sales by over 200 percent.


Follow us


Psychopathic or Just Antisocial? A Key Brain Difference Tells the Tale

Though psychopaths and antisocial people may seem similar, what occurs in their brains isn’t.

Common Knowledge Makes Us More Cooperative

People are more inclined to take mutually beneficial risks if they know what others know.

How a Shift in Human Head Shape Changed Everything

When did homo sapiens become a more sophisticated species? Not until our skulls underwent "feminization."

Journalists Can Get PTSD Without Leaving Their Desks

Dealing with violent content takes a heavy toll on some reporters.

Do Ticking Clocks Make Women More Anxious to Have Children?

Yes, but apparently only women who grew up poor.

The Big One

One in two full-time American fast-food workers' families are enrolled in public assistance programs, at a cost of $7 billion per year. July/August 2014 fast-food-big-one
Subscribe Now

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.