Transformer: Attention History May Matter

Transformer has shown to be a very effective finding to solve numerous learning tasks for various application fields, such as the image captioning task in which this work will focus. Its widespread success is owed to two main ingredients: 1) attention mechanism; and 2) positional encoding. This paper is interested in the first ingredient, showing that the vanilla attention mechanism may be improved by exploiting not only the context conveyed in a data sequence under analysis, but also in all sequences forming the complete training dataset. In particular, we introduce the concept of attention history to better capture and model contextual information. Two different strategies to compute attention history before its injection in the attention mechanism are described. The proposed solution is validated and discussed on four reference remote sensing image captioning datasets.