Beyond Surface Metrics: Modeling Abstractivity in Summarization via Action-Based Spectrums

Traditional views of abstractive summarization treat it as a free-form rephrasing of the source text, yet this notion is vague and difficult to measure; existing evaluations rely on surface-level metrics — such as novel n-gram counts, extractive fragment coverage, or extractive fragment density — that obscure the stylistic nuances that distinguish summaries along an abstractivity spectrum. We therefore propose a fine-grained, continuous characterization that unifies extractive and abstractive summarization as two endpoints of a single spectrum defined by the frequency of text-transformation actions (sentence reduction, combination, paraphrasing, reordering, etc.) drawn from the foundational summarization literature. To operationalize this view, we introduce CLAsum, a human-annotated dataset comprising 600 unique samples; these arise from 200 document–summary pairs sourced from CNN/DailyMail and XSum datasets, each annotated by three independent annotators who assign one of five perceived abstractivity levels and record the prevalence of seven abstractivity-inducing actions per pair. Our analysis demonstrates a strong correlation between these action frequencies and human judgments, validating the spectrum, while automatic prediction of the resulting Abstractivity-Inducing Features (AIFs) captures signals that existing metrics miss and yields significant gains in abstractivity-level measuremnt. This work delivers both conceptual clarity and practical tools for building controllable, style-aware summarization systems capable of producing summaries with precisely specified levels of abstractivity, thereby advancing the field toward truly user-tailored, task-adaptive summarization.