Integration of Multi-Modal Data into Generation of Text Descriptions: Methods, Challenges and Prospects

Current systems of AI more and more often use multi-modal data by combining visual, text and audio information to resolve complicated problems. One key sphere of such system application is description generation on the basis of images and video. Integration of multi-modal data provides an opportunity to improve accuracy and expressiveness of texts being created, which gives more complete and sensible representation of the content. The article studies current methods of multi-modal data integration into generation of text descriptions, analyzes key challenges that face researchers and discusses promising trends in this field of development. Special attention is paid to application of convolution neuron nets (CNN) and transformers to process visual information, as well as attention mechanisms and models of successive text generation. The article researches approaches to data fusion formed by different modalities, including earlier and later combination of signs and multi-modal models trained on big blocks of data. In spite of serious progress, integration of multi-modal data provokes a number of challenges, including information synchronization, problems in interpretation and context, restrictions in learning data and others. Promising lines in development are being discussed. Obtained results can be used by developers of computer vision systems, in processing natural language and multi-modal machine learning and for elaboration of intellectual applications in the field of automatic image abstracts, video-summarizing and man-machine interaction.