通过筛选关键注意力头，让大模型更高效地注入任务嵌入。
Soft Head Selection for Injecting ICL-Derived Task Embeddings
- 基于梯度选择与任务相关的注意力头，实现精准嵌入注入。
- 在12个不同规模模型上表现超越传统方法，参数量远低于微调。
- 揭示注意力头功能具有强任务依赖性，为模型机制提供新洞察。
大型语言模型通常通过参数高效微调（PEFT）或上下文学习（ICL）适配下游任务。近期提出的基于嵌入的ICL适配范式，从少量提示的中间激活中提取任务特定嵌入，并在推理时注入。尽管概念上有吸引力，但该方法在实际中性能提升有限，未能持续优于PEFT或ICL。本文提出软头选择方法（SITE），一种基于梯度的策略，用于识别任务相关的注意力头，以实现有效的任务嵌入注入。在开放生成、推理和自然语言理解等多种任务上，SITE显著优于先前的嵌入式适配方法和少样本ICL，同时使用远少于PEFT的可训练参数。对12个参数量从4B到70B的LLM的实验验证了方法的普适性；任务内与任务间激活修补分析进一步揭示注意力头功能具有强任务依赖性，提供了新的机制性见解。
原文摘要 · Abstract (English)
Large language models (LLMs) are commonly adapted to downstream tasks using parameter-efficient fine-tuning (PEFT) or in-context learning (ICL). Recently, ICL-driven embedding-based adaptation has been proposed as a distinct task adaptation paradigm. It derives task-specific embeddings from intermediate activations using few-shot prompts and injects them during inference. Despite its conceptual appeal, this approach has not demonstrated consistent performance gains over PEFT or ICL, and its empirical advantages have been limited in practice. We propose Soft head-selection for ICL-derived Task Embeddings (SITE), a gradient-based method that identifies task-relevant attention heads to enable effective task embedding injection. Across various types of open-ended generation, reasoning, and natural language understanding tasks, SITE significantly outperforms prior embedding-based adaptation methods and few-shot ICL, while using substantially fewer trainable parameters than PEFT. Experiments on 12 LLMs ranging from 4B to 70B parameters demonstrate the generality of our approach, and intra-task and inter-task activation patching analyses further provide new mechanistic insights by revealing strong task dependence in attention head functionality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。