改进函数向量生成方法，让大模型指令控制更高效准确
Fast & Faithful Function Vectors
- 用梯度归因结合LRP选择注意力头，提升效率与精度
- 分布式应用函数向量比简单聚合准确率更高
- 适合需要精准指令控制的LLM应用开发人员
函数向量（FVs）是上下文学习中提取的任务表征，可用于引导大语言模型（LLMs）。然而其构建方式的设计选择仍不明确。本文研究了沿两个自由度变化的FV定义对指令的影响：注意力头选择和向量引导方式。在头选择方面，采用基于梯度的归因方法结合层间重要性传播（LRP）显著提升了效率与准确性；在函数向量引导方面，分布式应用相比简单聚合具有更高准确率。代码已公开。
原文摘要 · Abstract (English)
Function vectors (FVs) are task representations elicited during in-context learning that can be used to steer Large Language Models (LLMs). However, design choices in their formulation remain underexplored. In this work, we study the impact of varying FV definitions for instructions along two degrees of freedom: attention head selection and steering. For head selection, using gradient-based attributions with Layer-wise Relevance Propagation (LRP) substantially improves efficiency as well as accuracy. For FV steering, applying it in a distributed manner yields a higher accuracy compared to simple aggregation. Our code is publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。