Using vision-language models for semantic search of a graphical user interface fragment on request

View at publisher Summarise with VIRAT
Authors: Кристина Дербина, R. A. Simonov
Cited by: 0
References: 11

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

This work analyzes the current state of modern methods for semantic search of graphical interface elements based on text queries in the context of highly variable visual representations of interfaces. Examined the influence of visual and textual metrics on the accuracy of query source analysis for functional interface elements. Found that the emergence of linear approaches based on independent changes in visual and textual representations does not contribute to the accurate consideration of the functional context of interface elements. Also found that the main limitation of such approaches is the lack of a mechanism for the mutual exchange of text and visual information at the parliamentary level. Developed architecture of a multimodel embedder of contextual information from technical documentation, which can be used in semantic search systems focused on the graphical interface elements, to increase the informativeness of text descriptions of interface elements. The proposed solution is based on the combined processing of the interface, element fragment, and textual context. Investigated an experimental approach to establishing a connection between elements of information system interface images and their semantic representations in technical documentation.

Fields of Study & Subjects

Multimodal Machine Learning Applications Software Engineering Research Information Retrieval and Search Behavior

Publication details

Journal / source
International Conference on Optics, Computer Applications, and Materials Science (CMSD)
Pages
21–21
Publication date
1 April 2026
Language
EN