A novel approach to document similarity retrieval using sentence transformers and vector databases

This study introduces a novel method for document similarity retrieval, leveraging Sentence Transformers for efficient processing and Milvus for vector storage. The workflow starts by extracting text from crowd-sourced vector databases and segmenting it into individual sentences. These sentences are transformed into embeddings using Sentence Transformer, creating a robust text representation. The embeddings are stored in Milvus, facilitating high-performance similarity searches. To improve query relevance, we enhance user queries with synonyms from WordNet, addressing different spellings and related terms. Our approach effectively tackles duplicate detection and spelling variations through vector similarity measures and customized indexing, ensuring accurate retrieval and ranking of relevant documents.