Hierarchical Attention Modeling for Handwritten Prescription Recognition: A Swin Transformer-Based Solution

Many doctors’ prescriptions are handwritten and often unreadable by the chemist who sells the medicine. Patients have been put in jeopardy as a result. A deep learning architecture based on the Swin Transformer for handwritten medical prescription identification has been proposed in the research paper. The proposed architecture has both a local and a global aspect. To identify character features, the local part uses handwriting features and contextual patterns through a hierarchical attention mechanism. The recommended framework is very easy to comprehend for both the handwriting style and the distortion. The BD dataset has 4,680 labeled images showing 78 different words associated with medicine. Handwritten prescriptions from doctors are included. The dataset also allows for the creation of a comprehensive preprocessing model that scales, normalizes, and applies sophisticated data augmentations that simulate real-world conditions. The test accuracy obtained was 89.0% with macro and weighted F1 scores of 0.88, performing better than some existing techniques involving CNN and CRNN. In addition, the results of the confusion matrix validate the model's ability to detect similar drug names. To demonstrate the efficiency of the model in segregating illegible handwriting from well-written handwriting, a qualitative study of the model’s performance was undertaken. The researchers showed that using the transformer model, they could effectively digitize handwritten prescriptions, which can successfully curb the errors associated with the selling of medicine. The study on the semantic understanding of the obtained results through the application of optical character recognition and language models via multi-modal fusion is a future direction for research.