Identifying Key Terms in Prompts for Relevance Evaluation with GPT Models

Relevance evaluation of a query and a passage is essential in Information Retrieval (IR). Recently, numerous studies have been conducted on tasks related to relevance judgment using Large Language Models (LLMs) such as GPT-4,demonstrating significant improvements. However, the efficacy of LLMs is considerably influenced by the design of the prompt. The purpose of this paper is toidentify which specific terms in prompts positively or negatively impact relevanceevaluation with LLMs. We employed two types of prompts: those used in previousresearch and generated automatically by LLMs. By comparing the performance ofthese prompts in both few-shot and zero-shot settings, we analyze the influence ofspecific terms in the prompts. We have observed two main findings from our study.First, we discovered that prompts using the term ‘answer’ lead to more effectiverelevance evaluations than those using ‘relevant.’ This indicates that a more directapproach, focusing on answering the query, tends to enhance performance. Second,we noted the importance of appropriately balancing the scope of ‘relevance.’ Whilethe term ‘relevant’ can extend the scope too broadly, resulting in less precise evaluations, an optimal balance in defining relevance is crucial for accurate assessments.The inclusion of few-shot examples helps in more precisely defining this balance.By providing clearer contexts for the term ‘relevance,’ few-shot examples contributeto refine relevance criteria. In conclusion, our study highlights the significance ofcarefully selecting terms in prompts for relevance evaluation with LLMs