On Disentanglement and Factual Consistency in Automatic Dialogue Summarisation
Loading...
Citations
Altmetric:
Altmetric::
Download
Associated Records
Authors
File Licence
Editors
Date
2024
Source Title
Journal Title
Journal ISSN
Volume Title
Publisher
Melbourne Authors
Affiliation
Computing and Information Systems
Document Type
PhD thesis
Subject Code
Citation
Access Status
Open Access
DOI
Access URL
Open Access URL
Open Access at PMC
Linked Resource URL
NHMRC Grant Code
ARC Grant Code
Description
© 2024 Rongxin ZHU
Abstract
Automatically summarising dialogues, such as discussion boards, group chats, and meeting transcripts, can help us quickly grasp key ideas from large volumes of conversations.
This task presents unique challenges, as the summarisation models must navigate complex interaction patterns within dialogues and generate fluent, coherent, and factually consistent summaries that reflect the facts in the dialogues.
In this thesis, we investigate how to develop effective dialogue summarisation models. First, we look at how we can segment multi-party dialogues into distinct threads, an important preprocessing step for summarisation. We then explore how to evaluate the factual consistency of summaries, as factual errors are found to be prevalent in generated summaries. Lastly, we develop compact summarisation models by learning from large language models.
More specifically, to segment a multi-party dialogue into threads, we first implement a multi-task learning framework to incorporate dialogue history to predict the reply-to relations between utterances. This strategy outperforms prior methods when manually extracted features, such as word overlap and direct user mention, are absent. Following this, we explore the use of bipartite graphs to group linked utterances into threads. Unlike previous methods that do so greedily (that is, predict relations between utterances based on their best scores), our approach attempts to predict these relations by considering all utterances. We found that it has the potential to outperform previous methods, suggesting a promising direction for future research.
To assess the factual consistency of generated summaries, we first develop a two-dimensional error typology and create a benchmark dataset featuring fine-grained factual error annotations for summaries generated by a suite of dialogue summarisation models. Following this, we propose an unsupervised method to automate factual consistency annotation and find that it performs on par with state-of-the-art annotation models but requires fewer resources.
Lastly, to build compact dialogue summarisation models that are factually consistent, we distil symbolic knowledge from large language models by generating factually consistent and inconsistent summaries and apply contrastive learning to train smaller student models. Automatic and human evaluations show that our method improves the factual consistency of student models without compromising other quality aspects, such as fluency and informativeness.
Notes
Terms of Use
Keywords
Natural Language Processing, Deep Learning, Pre-trained Language Models, Large Language Models, Automatic Text Summarisation, Dialogue Summarisation, Factual Consistency of Summarisation, Conversation Disentanglement