用软提示替代文本示例，让大模型更准更快更稳定。
MHA-RAG: Improving Efficiency, Accuracy, and Consistency by Encoding Exemplars as Soft Prompts
- 将示例转为可学习的软提示，通过注意力头数量调节生成
- 在多个问答任务上提升20分，推理成本降低10倍
- 对示例顺序不敏感，适合低资源领域迁移
在数据有限的情况下适应新领域，对基础模型而言仍具挑战且计算开销大。尽管已有研究证明使用领域特定示例作为上下文演示有效，但我们探究纯文本表示示例是否是最高效、最有效且最稳定的方案。本文提出一种替代方法：将示例表示为软提示，并设计一种示例顺序无关的模型架构。为此，我们引入多头注意力检索增强生成（MHA-RAG），其注意力头数量作为简单超参数，控制不同任务下的软提示生成。在多个问答基准和模型规模下，MHA-RAG相较于标准RAG性能提升20点，同时推理成本降低10倍GFLOPs，兼具更高准确率与更强效率，且对示例顺序不变。
原文摘要 · Abstract (English)
Adapting Foundation Models to new domains with limited training data is challenging and computationally expensive. While prior work has demonstrated the effectiveness of using domain-specific exemplars as in-context demonstrations, we investigate whether representing exemplars purely as text is the most efficient, effective, and stable approach. We explore an alternative: representing exemplars as soft prompts with an exemplar order invariant model architecture. To this end, we introduce Multi-Head Attention Retrieval-Augmented Generation (MHA-RAG), a framework with the number of attention heads serving as a simple hyperparameter to control soft prompt-generation across different tasks. Across multiple question-answering benchmarks and model scales, MHA-RAG achieves a 20-point performance gain over standard RAG, while cutting inference costs by a factor of 10X GFLOPs-delivering both higher accuracy and greater efficiency, invariant to exemplar order.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。