Using vision-language models for semantic search of a graphical user interface fragment on request
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
This work analyzes the current state of modern methods for semantic search of graphical interface elements based on text queries in the context of highly variable visual representations of interfaces. Examined the influence of visual and textual metrics on the accuracy of query source analysis for functional interface elements. Found that the emergence of linear approaches based on independent changes in visual and textual representations does not contribute to the accurate consideration of the functional context of interface elements. Also found that the main limitation of such approaches is the lack of a mechanism for the mutual exchange of text and visual information at the parliamentary level. Developed architecture of a multimodel embedder of contextual information from technical documentation, which can be used in semantic search systems focused on the graphical interface elements, to increase the informativeness of text descriptions of interface elements. The proposed solution is based on the combined processing of the interface, element fragment, and textual context. Investigated an experimental approach to establishing a connection between elements of information system interface images and their semantic representations in technical documentation.
Fields of Study & Subjects
Multimodal Machine Learning Applications
Software Engineering Research
Information Retrieval and Search Behavior
Publication details
- Journal / source
- International Conference on Optics, Computer Applications, and Materials Science (CMSD)
- Pages
- 21–21
- Publication date
- 1 April 2026
- Language
- EN