Acquired Electronics360

Intellectual Property

AI Can Now Evaluate the Quality of Short Stories

21 August 2017

For years researchers have been pondering the idea that artificial intelligence (AI) may be able to understand and generate narratives. But can AI recognize a good story when it sees one?

Source: D4N13l3/DeviantArtSource: D4N13l3/DeviantArt

Scientists at Disney Research and the University of Massachusetts, Boston have taken steps towards solving this problem. The team has created neural networks that can evaluate short narratives. These AIs don’t mimic the professional judgment of literary critics, but the technology can predict which stories will appeal to a large population of readers.

"Our neural networks had some success in predicting the popularity of stories," said Boyang "Albert" Li, a research scientist at Disney Research. "You can't yet use them to pick out winners for your local writing competition, but they can be used to guide future research."

"The ability to predict narrative quality impacts on both story creation and story understanding," said Markus Gross, vice president at Disney Research. "To evaluate quality, the AI needs some level of understanding of the text. And if AIs are to create narratives, they need to be able to judge the quality of what they are producing."

The lack of large databases of stories has proved to be a stumbling block for training literary AIs, but the scientists found that Quora is a good data source. The answers on Quora are in the form of stories, and the AI measures a story’s popularity by the number of upvotes it has.

The researchers gathered almost 55,000 answers and developed an algorithm to classify them as stories or non-stories. They ended up yielding more than 28,000 stories with an average of 369 words.

The team set out to understand the stories' complex semantics. They looked for ways to represent the influence of story structures in neural networks because a sequence of events can interact to reveal character intentions.

The research developed an AI that evaluated separate regions of each story, including the question that prompted each. The team created a network that looked at the regions independently and another network that took a holistic view of how the meaning of the events and story regions emerged. In each case, the AIs made predictions of which texts would prove popular with readers.

In experiments, neural networks showed an 18 percent improvement over a baseline text evaluation system.

The team will present the findings August 23rd at the International Joint Conference on Artificial Intelligence in Melbourne, Australia. To learn more about this study, click here.

To contact the author of this article, email

Powered by CR4, the Engineering Community

Discussion – 0 comments

By posting a comment you confirm that you have read and accept our Posting Rules and Terms of Use.
Engineering Newsletter Signup
Get the Engineering360
Stay up to date on:
Features the top stories, latest news, charts, insights and more on the end-to-end electronics value chain.
Weekly Newsletter
Get news, research, and analysis
on the Electronics industry in your
inbox every week - for FREE
Sign up for our FREE eNewsletter


Date Event Location
12-16 Aug 2018 Vancouver, Canada
11-13 Sep 2018 Novi, Michigan
27 Sep 2018 The Reef, Los Angeles
03-05 Oct 2018 Boston, Massachusetts
26 Oct 2018 Old Billingsgate
Find Free Electronics Datasheets