Soft Head Selection for Injecting ICL-Derived Task Embeddings

Large language models (LLMs) are commonly adapted to downstream tasks using parameterefficient fine-tuning (PEFT) or in-context learning (ICL).Recently, ICL-driven embeddingbased adaptation has been proposed as a distinct task adaptation paradigm.It derives taskspecific embeddings from intermediate activations using few-shot prompts and injects them during inference.Despite its conceptual appeal, this approach has not demonstrated consistent performance gains over PEFT or ICL, and its empirical advantages have been limited in practice.We propose Soft head-selection for ICLderived Task Embeddings (SITE), a gradientbased method that identifies task-relevant attention heads to enable effective task embedding injection.Across various types of openended generation, reasoning, and natural language understanding tasks, SITE significantly outperforms prior embedding-based adaptation methods and few-shot ICL, while using substantially fewer trainable parameters than PEFT.Experiments on 12 LLMs ranging from 4B to 70B parameters demonstrate the generality of our approach, and intra-task and inter-task activation patching analyses further provide new mechanistic insights by revealing strong task dependence in attention head functionality.