A COMPARATIVE STUDY OF INFORMATION RETRIEVAL MODELS: BM25 VERSUS HYBRID RETRIEVAL ON THE CRANFIELD COLLECTION

Information retrieval systems face the fundamental challenge of balancing retrieval effectiveness with computational efficiency. Traditional lexical modelslike BM25 provide fast retrieval but may misssemantic relationships,while modern hybrid approaches combining lexical and semantic methods promise improved effectiveness at thecost of computational complexity. This study compares the performance of BM25 and hybrid retrieval modelson the Cranfield collection, evaluating both retrieval effectiveness and computational efficiency. We implementedtwo retrieval systems: (1) BM25 with pseudo-relevance feedback and tuned parameters (k1 = 1.6, b = 0.4), and(2) a hybrid model combining BM25, dense retrieval using SPECTER embeddings, Reciprocal Rank Fusion, andcross-encoder reranking. Both systems were evaluated on 225 Cranfield queries using standard IR metrics:Precision@10, Recall@20, Mean Average Precision (MAP), and NDCG@20. The hybrid model demonstratedsuperior retrieval effectiveness with 41% improvement in Precision@10 (1.07% vs 0.76%), 67% improvementin Recall@20 (1.54% vs 0.92%), and consistent gains across all metrics. However, BM25 showed dramaticallysuperior efficiency with 800x faster query processing (15.3ms vs 12.4s average latency). While hybrid retrievalmodels achieve better effectiveness, the computational cost may limit their applicability to real-time scenarios.The Cranfield collection’s inherent difficulty (1960s terminology, sparse relevance judgments) constrains absoluteperformance regardless of model sophistication.