Adversarial Contrastive Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) basically strengthens natural language generation by using large language models (LLMs) where the external knowledge is integrated to enable precise and context-aware responses. However, incorporating over-length retrieved passages substantially increases computational cost and response latency due to the quadratic complexity using transformer, especially when multiple passages are concatenated. To mitigate this issue, this paper presents a query-aware prompt compression framework that selectively preserves relevant tokens while discarding those redundant tokens. In particular, an adversarial contrastive learning is implemented to enhance the robustness of the compressed prompt against noisy or semantically similar but irrelevant passages, thereby improving the compressor’s ability to concentrate the tokens with sufficient evidence. Under the RAG setting, the proposed method assures efficient token utilization by shortening the context length and reducing the computation overhead, while the answer quality and the robustness to retrieval noise are preserved. Experimental results validate the merit of the proposed adversarial contrastive prompt compression in LLM-based natural language generation under retrieval-intensive tasks.