One Task Vector is not Enough: A Large-Scale Study for In-Context Learning
Abstract
In-context learning (ICL) enables Large Language Models (LLMs) to adapt to new tasks using few examples, with task vectors, defined as specific hidden state activations hypothesized to encode task information.Existing studies are limited by small-scale benchmarks, restricting comprehensive analysis.We introduce QUITEAFEW, a novel dataset of 3,096 diverse few-shot tasks, each with 30 input-output pairs derived from the Alpaca dataset.Experiments with Llama-3-8B and Qwen-3-8B on QUITEAFEW reveal: (1) task vector performance peaks at an intermediate layer, (2) effectiveness varies significantly by task type, and (3) complex tasks rely on multiple, subtaskspecific vectors rather than a single vector, suggesting distributed task knowledge representation.
Fields of Study & Subjects
Anomaly Detection Techniques and Applications
Publication details
- Journal / source
- Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)
- Publisher
- Association for Computational Linguistics
- Pages
- 1451–1463
- Publication date
- 1 January 2026
- Language
- EN