Sentence Vectors Enable GRU And Multi-Headed Attention-Based Video Captioning Framework For Specifying Events In MSVD Video Dataset

The fast growing digital culture has attracted people’s attention and motivated them to interact with fascinating multimedia materials, including images, videos, voice notes and messages. Video is the most often used digital material format globally; it is used for marketing for companies, education, proof of concern, memorial ceremonies, and amusement among other purposes. The primary goal of video/image captioning is to represent the dynamics of a video clip using simple and understandable conversational language. The motivation for captioning comes from its ability to enhance video accessibility for those who are deaf or hard-of-hearing, improve concentration and memory of information and allow viewing in sound-sensitive environments. In this proposed framework, we demonstrate how to create captions for videos using Gated Recurrent Unit (GRU) along with a Sentence Vector Branch (SVB) and skip connections employing two word embedding methods: Bidirectional Encoder Representations from Transformers (BERT) and Embeddings from Language Model (ELMO). The encoder section of the model uses skip connections to address the issue of vanishing gradients and the sentence vector branch is used to improve the efficiency of the encoder-decoder architecture. The decoder generates a suitable description using the sentence vector branch together with past generated words. All of our experiments make use of the Standard MSVD databases. BLEU@4, METEOR, ROUGE and CIDEr measures have been applied for the performance assessment. Comparative analysis of BERT and ELMO word embeddings reveals their respective strengths. Features are extracted from pre-trained CNNs NASNet-Large, VGG16, Inceptionv3, DenseNet-169 and ResNet50. In the MSVD dataset, BERT beats ELMO in the case of the GRU-based model, whereas ELMO beats BERT in the case of the LSTM-based model; among LSTM and GRU, LSTM outperforms GRU. DenseNet-169 beats every feature extraction method used in the MSVD dataset.