Advancing image captioning: Multilingual capabilities and Hindi-specific applications using transformer-based architectures with real-time smart focus camera for visually impaired users

This study focuses on image captioning, that is, creating a text description of images. It combines computer vision and natural language processing (NLP). One for how a computer sees the image and the other for how computer understands and generates text. This study introduces a new approach called Caption Transformer (CPTR) which is based on transformers instead of traditional CNNs(used for image processing) and RNNs(used for text generation). Avoiding these traditional methods, our approach improves efficiency and accuracy. The optimistic feature of this study is that this paper is using caption generation in the Hindi Language, addressing the challenges related to language diversity and cultural differences. The Hindi language is targeted because of the high audience availability than any other language. Additionally, this study integrate a Smart Focus Camera (SFC) designed to help visually impaired individuals. The camera automatically focuses on the center of the frame to detect objects (to focus on). The algorithm then takes the focus object and implements the captions in Hindi. It further converts the captions into voice assistance in real-time for visually impaired users.If no object is detected within the focus area of the camera, then the user will be assisted to look around through the voice assistance until the further object is detected. Our study explored multilingual training methods highlighting the power of self-attention mechanisms in transformers and suggest ways to improve Hindi captioning. By using specialised datasets and optimizing performance, our system aims to create accurate, meaningful, and real time captions to assist the visually impaired users.