Neural Networks and Knowledge Integration: Ontology-Driven Text Encoding for CLIP
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
This paper presents a method for improving the quality of multimodal "Text-Image" representations by integrating external ontological knowledge into a text query.The proposed approach integrates semantic and associative knowledge from ontologies directly into the text module of the CLIP neural network model through a tree-like syntax structure, similar to the K-BERT method.Adapting the syntax tree for use as an input parameter of the text module involves modifying the original positional encoding mechanism and attention mechanism.Relevant research was conducted on ways to modify the attention mechanism in accordance with the ontological nature of the syntax tree.The results of the experiments showed that the modified model outperforms the basic CLIP in terms of image classification accuracy in a number of object categories.A particularly noticeable increase in quality is observed for specific classes that require additional conceptual knowledge.The article discusses in detail the model architecture, the process of knowledge enrichment, and the impact of the proposed modifications on the model's performance.
Fields of Study & Subjects
Multimodal Machine Learning Applications
Advanced Neural Network Applications
Domain Adaptation and Few-Shot Learning
Publication details
- Journal / source
- Baltic Journal of Modern Computing
- Volume
- 14
- Issue
- 1
- Publication date
- 1 January 2026
- Language
- EN