A Systematic Comparison between Extractive Self-Explanations and Human Rationales in Text Classification

Instruction-tuned LLMs are able to provide an explanation about their output to users by generating self-explanations, without requiring the application of complex interpretability techniques.In this paper, we analyse whether this ability results in a good explanation.We evaluate self-explanations in the form of input rationales with respect to their plausibility to humans.We study three text classification tasks: sentiment classification, forced labour detection and claim verification.We include Danish and Italian translations of the sentiment classification task and compare self-explanations to human annotations.For this, we collected human rationale annotations for Climate-Fever, a claim verification dataset.We furthermore evaluate the faithfulness of human and selfexplanation rationales with respect to correct model predictions, and extend the study by incorporating post-hoc attribution-based explanations.We analyse four open-weight LLMs and find that alignment between self-explanations and human rationales highly depends on text length and task complexity.Nevertheless, selfexplanations yield faithful subsets of tokenlevel rationales, whereas post-hoc attribution methods tend to emphasize structural and formatting tokens, reflecting fundamentally different explanation strategies.