Automatic Story Generation and Evaluation
Loading...
Citations
Altmetric:
Altmetric::
Download
Associated Records
Authors
File Licence
Editors
Date
2024-05
Source Title
Journal Title
Journal ISSN
Volume Title
Publisher
Melbourne Authors
Affiliation
Computing and Information Systems
Document Type
PhD thesis
Subject Code
Citation
Access Status
Open Access
DOI
Access URL
Open Access URL
Open Access at PMC
Linked Resource URL
NHMRC Grant Code
ARC Grant Code
Description
© 2024 Zhuohan Xie
Abstract
Stories are created as a means to articulate our perspectives on the world, or offer an escape fromreality by providing an outlet for imagination. In natural language processing, studies have explored using machine learning models to generate stories, but this is a challenging task since it requires the story generation model to produce not only fluent and coherent narratives, but also interesting and creative content.
In this thesis, we concentrate on both the automatic generation and evaluation of stories. Initially, we construct story generation models using pre-trained language models. Next, we examine the proficiency of large language models in story generation, given their strong performance in various natural language generation tasks. Lastly, we focus on designing evaluation metrics capable of measuring different qualities, such as fluency and interestingness, of stories.
More concretely, we first propose a novel framework that combines two types of pre-trained language models, encoder based and decoder based, to build a variational autoencoder for story generation. The objective is to combine the merits from both worlds to create a story generation model capable of producing stories that are both high in quality and diversity. Over the past few years, the development of language models have significantly influenced the landscape of story generation. Observing that large language models have a good performance on many generation tasks, we conduct a comprehensive investigation, utilizing both automatic and human evaluation, to compare the story generation capacity of large language models with smaller pre-trained models. Our results demonstrate that large language models generate stories of significantly higher quality. Moreover, they exhibit a level of performance that is competitive with the quality produced by non-expert human crowd-workers. Finally, our last piece of work proposes a new evaluation framework that assesses multiple aspects of story quality such as fluency, relatedness and interestingness through perturbations. Extensive experiments show efffectiveness and robustness of our evaluation framework.
Notes
Terms of Use
Keywords
automatic story generation, automatic story evaluation, large language models, natural language generation, evaluation metrics, automatic text generation, human evaluation, variational autoencoder