A Hybrid-Based Deep Learning Framework for Pill Detection, Multi-Attribute Classification, and Prediction With Metadata Retrieval

Adverse Drug Events (ADEs) act as a significant challenge to preventable death and are the third leading cause of death in the United States, responsible for an estimated 250,000 to 300,000 deaths annually. This study presents a hybrid deep learning method for automated pill detection, classification, and prediction to reduce medication errors and support older adults, users with visual impairment, and people with limited literacy. The proposed framework is a combination of a deep learning model, YOLOv12n, to detect a pill, a ConvNeXt-Tiny computer vision model, which can classify the shape, color, and class of the pill, and a metadata verification component to make a final prediction. A manually labeled dataset of 200 pill classes, each with 50 images (10,000 images), each image containing two pills (front and back), for a total of 20,000 pills, was developed to support training and validation, including both tablets and capsules. Five lightweight YOLO variants from YOLOv8n to YOLOv12n were evaluated in true zero-shot and supervised learning settings using the AdamW optimizer. YOLOv12n has shown better results in supervised mode, with a precision of 0.972, a recall of 0.995, and a mAP50 of 0.990. ConvNeXt-Tiny achieved the highest classification accuracy of 98.68% for shape, 100% for color, and 99.32% for class. The prediction uses a hybrid framework (YOLOv12n + ConvNeXt-Tiny + Metadata) that outperforms the baseline approach (YOLOv12n + EfficientNet-B0 + Metadata), achieving 0.99 for class, 0.999 for shape, and 0.998 for color. The reliable development of the confusion matrices confirmed that the predictions were consistent and highly confident, suggesting that the hybrid system shown improved accuracy, robustness, and interpretability, and therefore represents a promising AI-based solution for pill identification and ADE avoidance in a real-time scenario.