A hybrid framework for pill identification using convolutional neural networks and optical character recognition

Medication errors caused by pill identification errors result in more than 100,000 deaths annually, according to the figure provided by the World Health Organization. It outlines and discusses the proposal of a method to develop a hybrid deep learning approach that uses separately Convolutional Neural Networks (CNN) and Optical Character Readers (OCR) in pill image recognition and imprint detection before being individually evaluated, comparatively analyzed, and finally combined using an ensemble method. We implemented images from datasets provided by the Ministry of Food and Drug Safety (MFDS), in collaboration with the National Library of Medicine (NLM), and even an image dataset of pills from the ePillID dataset in low-shot scenarios. Our hybrid approach reaches top-1 accuracy ratings of 87.2%, 76.8% and 81.5% upon being used in ensemble mode from images sourced from MFDS and reference images from NLM and consumer-grade images sourced from NLM, exceeding current top-1 accuracy ratings of 82.1% and 78.4% in standalone image evaluations by CNN and OCR, respectively. Our image analysis and processing takes an average of 0.65 s, making it even more suitable for mobile devices. Our approach is significant in that it develops and creates a paradigm through which pill images and imprints can be individually and comparatively evaluated and integrated for adaptive ensemble evaluation. This work introduces a modular hybrid CNN–OCR ensemble that treats visual recognition and imprint extraction as fully independent, comparatively evaluated modules before applying an adaptive, per-class weighted fusion. Unlike recent end-to-end or fixed-fusion approaches, our framework explicitly quantifies each module’s strengths and weaknesses, yielding a statistically significant 5.1% absolute top-1 accuracy improvement (95% CI [3.8-−6.4%], paired t-test \(p < 0.01\) ) while maintaining 0.65 s mobile inference—making it particularly suitable for real-world polypharmacy safety and assistive applications.