无需微调,用上下文学习构建解码空间任务向量,提升大模型表现。
DeCoVec: Building Decoding Space based Task Vector for Large Language Models via In-Context Learning

- 基于上下文学习在解码空间生成任务向量,不修改模型权重。
- 在多个数据集上最高提升5.50%准确率,优于标准少样本基线。
- 抗提示顺序干扰,抑制生成退化,适合快速部署的场景。
任务向量作为编码任务特性的方向,在模型或激活空间中被用于引导大语言模型(LLM)。然而,现有方法通常需要微调或侵入性地修改内部状态,限制了灵活性与可扩展性。本文提出 extsc{DeCoVec}(基于解码空间的任务向量),一种无需训练、非侵入性的框架,通过利用上下文学习(ICL)直接在解码空间构建任务向量。具体而言, extsc{DeCoVec} 将少量示例与零样本提示输出概率分布之差作为任务本质,注入解码过程以引导生成。在七个规模为0.5B至9B的LLM上,于TruthfulQA、Math-500和AQUA-RAT三个数据集上的实验表明, extsc{DeCoVec} 始终优于标准少样本基线,平均准确率最高提升+5.50。进一步分析显示,该方法有效抑制生成退化与逻辑错误,且对示范顺序具有强鲁棒性,同时不增加额外输入标记成本。本方法提供了一种无需权重更新或辅助模型的训练自由、非侵入式的大模型引导方案。
原文摘要 · Abstract (English)
Task vectors, representing directions in model or activation spaces that encode task-specific behaviors, have emerged as a promising tool for steering large language models (LLMs). However, existing approaches typically require fine-tuning or invasive manipulation of internal states, limiting their flexibility and scalability. We propose \textsc{DeCoVec} (Decoding Space based Task Vector), a training-free and non-invasive framework that constructs task vectors directly in the \textit{decoding space} by leveraging in-context learning (ICL). Specifically, \textsc{DeCoVec} captures the task essence as the difference between the output logit distributions of few-shot and zero-shot prompts, then steers generation by injecting this vector into the decoding process. Experiments across seven LLMs (0.5B--9B) on TruthfulQA, Math-500, and AQUA-RAT show that \textsc{DeCoVec} consistently outperforms standard few-shot baselines, with gains up to +5.50 average accuracy. Further analysis demonstrates that \textsc{DeCoVec} effectively suppresses generation degeneration and logical flaws while exhibiting strong robustness to demonstration ordering, all without incurring additional input token costs. Our method offers a training-free and non-invasive solution for LLM steering without requiring weight updates or auxiliary models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。