Interpretable Document-Level Polarity Classification with Inter-sentence Attention

Large language models (LLMs), such as BERT and GPT-3, significantly impact our daily lives.They can derive accurate outputs for many tasks about natural languages, but they do not explain the reason for the outputs.In this sense, LLMs work as a black box and have the problem of interpretability.This paper is devoted to considering polarity classification of documents.Compared to simple sentence-level polarity classification, document-level one makes it more difficult to create a high-performance model because we have to consider relationships between sentences.To tackle this change, we use inter-sentence attention, which can capture the relationship between sentences: the higher an inter-sentence attention score is, the more relevant the corresponding sentences are to each other.We use intersentence attention scores to capture the context of sentences and propose a model whose classification is more similar to human judgment.To validate the proposed model, we conduct three types of experiments: one is to compare classification performance with prior models; the second one is to compare interpretability with prior models; and the last one is to show the ability of inter-sentence attention whether it could capture the relationship between sentences.From the first two types, we found that our model is more accurate on two real datasets.In the second type of experiment to assess interpretability, we examined the overlap between sentences that contribute to the model's predictions and those annotated by humans for the same document and found that our model has a larger overlap and is more likely to extract interpretive sentences that humans intuitively consider important.In addition, our result partially captures the polarity of "implicit" sentences that do not contain direct expressions, which could not be captured by prior models, suggesting that our model may lead to a more natural interpretation.From the third type of experiment, we show that our model can capture the contexts of sentences.