DATA DETECTIVES: ARE YOU LESS CREATIVE THAN CHATGPT?

By Connor Zahler

EVERYONE’S SO CREATIVE

Over the past year, it’s been impossible to go a single day without a breathless headline about the promise (or doom) of AI technologies. One of the hobby horses in this cottage industry is a focus on jobs. AI will destroy your job. AI is going to make your job easier. AI will take your job, but you might get a different one from it. In most of these articles, the jobs at risk are generally clerical in nature and require doing the same thing over and over again. Routine tasks seem a lot easier to program into an algorithm than anything else. Wait, though! A new article says that creative jobs aren’t safe either.

Recently, the Wall Street Journal published an article titled “M.B.A. Students vs. ChatGPT: Who Comes Up With More Innovative Ideas?” Based on a recent study by Wharton School professors, the article has some bad news for the creative class: ChatGPT can come up with better ideas than you can. Could this be true? Is it over for every profession? Let’s dive into both the article and the study to find out.

THE ARTICLE

The WSJ article was published by two of the four authors of the working paper it’s based on, those being Christian Terwiesch and Karl Ulrich. They explain that AI is conventionally understood to be bad at creative tasks, an idea they wanted to put to the test. To do so, they asked students in courses they were teaching on innovation to come up with some original ideas for products to sell to college students for less than $50. From all these responses, they randomly selected 200 to be compared to 200 ideas from ChatGPT (100 based on a prompt, 100 based on training with previous responses that had done well). These responses were put to a panel, who rated the ideas on both how much they would want to buy it and how novel they thought the idea was.

In short, the results were a massacre in favor of ChatGPT. The average purchase intent for human-generated ideas was 40%, while ChatGPT’s was 47%, a value that rose to 49% with training. Of the 40 most favored ideas from the whole dataset, 35 were from ChatGPT. The authors feel that these results indicate that generative AI is set to become a major force in business innovation, one that should be consulted alongside any human efforts. In the end, humans will move from being idea generators to being idea evaluators, with the ideas coming from generative AI. 

From these results, it seems like the case should be closed: ChatGPT comes up with better ideas than people, especially when given some training. It isn’t our way to answer a question that quickly, though. To really get to the bottom of this, we’ll have to delve into the working paper.

THE WORKING PAPER

First, it’s important to note that this study has not yet been formally published in an academic journal: the version accessible through the WSJ article is still listed as a working paper. For this paper, Terwiesch and Ulrich were joined by Lennart Meincke (also of Wharton) and Karan Girotra (of Cornell). Most of the working paper is just a further explication of what they say in the article, but there are two things worth mentioning before getting into the most important part: the difference in purchase intent between ChatGPT and humans was found to be statistically significant, and the difference in novelty between the groups was not found to be significant. Within the confines of the study, it seems like the right conclusion is that ChatGPT is better at generating innovative ideas.

…is what I would say, if the authors had not included a list of the 40 top ideas mentioned earlier. This list of ideas allows you to see what people were actually saying they want to buy, rather than just abstract groups. What are some of these ideas? Here’s a selection of ChatGPT’s ideas:

  • “Compact printer”
  • “Noise-Canceling Headphones”
  • “Reusable Silicone Food Storage Bags”
  • “CollegeLife Collapsible Laundry Hamper”
  • “Dorm Room Air Purifier”

What do these ideas have in common? Most of them already exist or are completely impractical to sell for less than $50 dollars (looking at you, compact printer). This isn’t exclusive to these ideas, either. When looking through the list of 40, it’s hard to find something that, as a college student, I don’t own, or know someone who does own.

The authors do clarify that they were not attempting to test novelty based on the prompt provided. There’s a difference between the degrees of novelty for an idea and whether or not something already exists, though. A lot of the ChatGPT-generated ideas are things I could go to Target and buy right now. The only innovation on display is offering the product for less than $50, with no care given to how feasible that may or may not be.

For the sake of comparison, here’s the five human ideas:

  • “Adaptiflex [cord extension to fit big adapters]”
  • “Kitchen Safe Gloves”
  • “Bluetooth Signal Merger [share music]”
  • “Smartphone Projector”
  • “Grocery Helper [hook to carry multiple bags]”

You’ll notice that these ideas are generally a lot more modest in their ambitions, which makes sense, given the $50 limitation. They’re also not terribly original, but not any less than the ChatGPT ideas, and they’re a lot more feasible.

It should also be noted that the ideas came with a short explanation or summary, which is not presented in the working paper. Sure, it’s possible that these summaries contained some kernel of information that turned the idea into something entirely new and different, but the authors apparently didn’t find them important enough to include in the paper. From the headlines they give, the ChatGPT ideas don’t seem innovative or even feasible. 

MORAL OF THE STORY

At a basic level, this study showed that ChatGPT could come up with ideas that consumers were more likely to want to purchase than those generated by humans. Those ideas, though, often already existed or face massive practical barriers. We already know that ChatGPT is great at regurgitating information that already exists or providing fantastical ideas. To say that ChatGPT is already much more creative and innovative than humans based on this study ignores every consideration other than what one online panel found. The fact that these ideas need to be refined and polished doesn’t necessarily refute the premise that humans will move towards a more evaluatory role in idea generation. It bears asking, though, whether the work required to hone and refine AI ideas is that much less than what we already do with human ideas.

If you’re in a creative field, you don’t need to panic just yet. Even if AI takes over idea generation, there’s clearly still a lot of work required to get those ideas to a place where they’re actually new or innovative. For those hoping that ChatGPT will give them a new million dollar idea, be advised: you might just be reinventing the wheel.