A study of handwritten text recognition with cross-out words

Abstract Handwritten Text Recognition (HTR) is a challenging task in computer vision and the presence of crossed-out words in handwritten documents significantly impacts the recognition process. In this study, we first analyze the performance of three HTR methods on seven types of cross-outs. We compared HTR models based on Convolutional Recurrent Neural Networks (CRNN), Sequence-to-Sequence (Seq2Seq) architectures, and a Character Spotting (CSpot) approach. We then investigate two strategies to improve recognition of crossed-out words. The first explores improving the HTR models’ performance on different styles of cross-outs by adding crossed-out samples to the training data. The second explores improving HTR model performance by using a cross-out removal method to clean the handwriting. For our experiments, we use word images from the IAM dataset with synthesized cross-outs, as well as a newly created handwritten word image dataset with real cross-outs. The results show that both strategies significantly improve recognition accuracy; however, the best strategy depends on the context. CSpot performed better when trained with crossed-out samples. Cross-out cleaning works best for CRNN and Seq2Seq architectures, but the data requirement to implement this for real collections means it might not be practical for use. Datasets and code will be publicly available.