Beyond Pointwise: Exploring Comparative Approaches for Predicting Query Performance Using LLMs

Query Performance Prediction (QPP) is a technique for evaluating the retrieval quality of a search system for a given query without relying on human relevance judgments. Recently, Large Language Models (LLMs) presented a promising avenue for this task. However, while existing pointwise methods have proven effective, they do not fully exploit the potential advantages offered by comparative analyses, such as pairwise comparisons between retrieved elements. Our study evaluates the suitability of these approaches for post-retrieval, LLM-based QPP. We conducted a comprehensive analysis comparing LLMs’ (Llama-3.1-8B-Instruct and Qwen3-8B) outputs with human-judged performance using correlations of metrics such as NDCG@10 and RR@10 and assessed computational efficiency regarding the number of LLM inferences. When evaluated on the correlation between actual retrieval performance and performance predicted by LLM methods, nearly all pairwise approaches outperform the pointwise baseline in the setting without fine-tuning, and in selected fine-tuning scenarios. Furthermore, pairwise approaches show substantially higher correlation with the ANCE retriever, while their correlation with BM25 varies across methods, with some exhibiting stronger alignment and others performing similarly or worse than the pointwise method. Our approach reduces computational complexity by introducing comparisons only against a selected subset of reference items.