Efficient and Effective In-context Demonstration Selection with Coreset

In-context learning (ICL) has emerged as a powerful paradigm for Large Visual Language Models (LVLMs), enabling them to leverage a few examples directly from input contexts. However, the effectiveness of this approach is heavily reliant on the selection of demonstrations, a process that poses significant challenges due to its NP-hard nature. Traditional strategies, including random, similarity-based sampling and infoscore-based sampling, often lead to inefficiencies or suboptimal performance, struggling to balance both efficiency and effectiveness in demonstration selection. In this paper, we propose a novel demonstration selection framework named Coreset-based Dual Retrieval (CoDR). We demonstrate that samples within the diverse subset achieve higher mutual information expectations. To implement this, we introduce a cluster-pruning method to build a diverse coreset. This coreset aligns more effectively with the input query while maintaining diversity. Additionally, we introduce a dual retrieval mechanism to enhance the selection process, achieving a more global demonstration selection, while maintaining efficiency. Experimental results demonstrate that our method significantly improves the ICL performance compared to the existing strategies, providing a robust solution for effective and efficient demonstration selection.