The Continued Value of Classic Summarization Models: Boosting Performance With High-Quality References

Recent studies indicate a growing preference for summaries generated by large language models (LLMs) over those produced by classic neural network-based models, highlighting a significant performance gap. We hypothesize that this disparity may arise from the quality of the reference summaries used to train these models. To explore this, we first conduct an evaluation comparing conventional ground truth references with summaries generated by LLMs. Following this, we design a series of experiments to assess the performance of classic models trained on higher-quality references. Surprisingly, our findings confirm that classic models trained with improved references can achieve performance comparable to LLM-generated summaries, as validated by both human evaluations and automatic metrics. Our results emphasize that classic summarization models retain distinct advantages, including task-specific design, smaller size, and ease of training. Additionally, we introduce GXSum1, a novel dataset aimed at fostering advancements in text summarization research. This work underscores the potential of leveraging LLMs not only as tools for summary generation but also as resources for creating enhanced training data, paving the way for improved performance in classic summarization models.