Linguistically-Inspired Neural Coherence Modeling

Coherence is an essential property of well-written text, making it easier to read and understand than a sequence of randomly arranged sentences. Assessing text coherence is valuable for many tasks. For example, it can be used to automatically score documents, reducing manual effort, or to provide feedback to students, helping them improve their writing quality. It can also serve as a reward model for training Large Language Models (LLMs) to generate more coherent and natural text. Given the importance of the task, many methods have been proposed for coherence modeling. Among these approaches, the dominant ones are neural network-based models due to their strength in representation learning and feature combination. In linguistics, many factors contribute to achieving textual coherence. For example, text coherence can be achieved by describing the same set of entities or using discourse relations between sentences. However, existing work on neural coherence modeling focuses on using more powerful encoders or solely entity information, without a systematic analysis of the benefits of different linguistic features. In this thesis, we investigate the importance of entity- and relation-based patterns for coherence assessment and develop novel approaches to utilize these features individually or jointly. We first investigate the benefits of entity-based patterns for coherence modeling. We analyze previous work that has leveraged entity patterns for coherence assessment. Then, we introduce a novel graph-based approach that captures the similarity of entity transition patterns between documents, rather than limiting the modeling of these patterns within a single document. We evaluate this approach on multiple benchmarks, and the results demonstrate that it outperforms various baselines. Next, we examine the role of discourse relations in coherence modeling. Existing discourse parsers struggle with implicit discourse relation classification, limiting the use of discourse relations in coherence assessment. To address this, we propose a novel framework that jointly generates a connective between arguments and predicts discourse relations based on both the arguments and the generated connectives. Experiments show that our joint model achieves state-of-the-art performance on the PDTB 2.0, PDTB 3.0, and PCC datasets. Beyond proposing a novel model for implicit discourse relation classification, we also investigate an unanswered question in the discourse processing community: why do relation classifiers trained on explicit examples (with connectives removed) perform poorly in real implicit scenarios? We identify label shift caused by the removal of connectives as a key factor contributing to this failure. To support this finding, we provide both manual analysis and corpus-level empirical evidence. Additionally, we propose two strategies to mitigate the impact of label shift. Using the improved discourse parser, we identify discourse relations within documents and empirically demonstrate their correlation with textual coherence. Based on this observation, we develop a novel fusion model that integrates discourse relation-based features into a pre-trained model for coherence modeling. Finally, we explore combining entity-based and discourse relation-based features for coherence modeling. This approach is motivated by the observation that writers typically employ multiple strategies simultaneously to ensure coherence. To this end, we design two methods to jointly model entities and discourse relations for coherence assessment. Experimental results demonstrate that both approaches significantly outperform models that use either features in isolation, highlighting the importance of considering both types of features simultaneously.