Joint Optimization Framework Combining Sparse Representation and Dense Retrieval in PubMed Retrieval

The exponential growth of biomedical literature indexed in PubMed has necessitated thedevelopment of advanced information retrieval systems capable of navigating complexterminologies and user intents. Traditional sparse retrieval models, such as BM25, excel atexact lexical matching but fail to capture semantic nuances, whereas emerging denseretrieval techniques utilizing transformer-based encoders offer semantic understanding butoften struggle with precise entity matching and out-of-vocabulary domain-specific terms.This paper proposes a novel framework that jointly optimizes sparse representations anddense retrieval vectors within a unified learning objective. By integrating a sparsityinducing regularization term into the dense encoder training phase, our model learns toretain critical lexical features while simultaneously projecting documents into a semanticvector space. We demonstrate that this hybrid approach mitigates the vocabulary mismatchproblem inherent in biomedical queries without requiring heavy post-hoc reranking stages.Extensive experiments conducted on standard biomedical information retrieval benchmarksreveal that the proposed framework significantly outperforms strong baselines, offering arobust solution for precision-critical medical search tasks. The results suggest that thesynergistic combination of lexical exactness and semantic flexibility is essential for the nextgeneration of biomedical search engines.