Optimizing sentence retrieval with dual embedding strategies: A hybrid BERT and Word2vec approach

Semantic search has proved to be a significant development over basic keyword-based search processes by employing contextual knowledge of searches. We developed and deployed a hybrid semantic search engine integrating the distributional semantics of Word2Vec with the contextual strength of BERT (Bidirectional Encoder Representations from Transformers). The models take a dual embedding strategy and utilize FAISS-based similarity search in addition to a fusion approach to reconcile relevance scores of both embeddings. To facilitate easier retrieval within this dual-embedding system, the system employs FAISS, which is a high-speed library for fast approximate nearest neighbor (ANN) search. FAISS enables efficient similarity searching through huge collections of high-dimensional vectors. Individual FAISS indices are maintained for the Word2Vec and BERT embeddings. This paper, we present a hybrid semantic search system that blends the strength of two powerful models, Word2Vec, and BERT (Bidirectional Encoder Representations from Transformers), each model represents linguistic information differently better.