用运放机制增强大模型对关键信息的注意力，提升问答准确率
Efficient OpAmp Adaptation for Zoom Attention to Golden Contexts
- 借鉴运放原理设计注意力适配器，高效聚焦核心信息
- 在噪声上下文测试中超越DeepSeek-V3和GPT-4o
- 无需从头训练，适配器可无缝集成到现有模型
大语言模型在问答任务中表现优异，尤其在检索增强生成（RAG）和长上下文场景中。然而，其性能常受噪声参考文档干扰，难以有效关注关键信息。尽管已有微调尝试，基于Transformer的架构仍倾向于分配过多注意力给无关或靠后文档。近期提出的差异注意力机制受限于不合适的共模抑制比（CMRR）及高计算成本。受运算放大器（OpAmp）启发，我们提出OpAmp适配机制，通过高效适配器实现。将适配器嵌入预训练Transformer模块，可在不进行大规模训练的前提下增强对黄金上下文的关注。在含噪声上下文的基准测试中，使用该机制训练的Qwen2.5-OpAmp-72B模型性能优于当前最先进模型，包括DeepSeek-V3和GPT-4o。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown significant promise in question-answering (QA) tasks, particularly in retrieval-augmented generation (RAG) scenarios and long-context applications. However, their performance is hindered by noisy reference documents, which often distract from essential information. Despite fine-tuning efforts, Transformer-based architectures struggle to prioritize relevant content. This is evidenced by their tendency to allocate disproportionate attention to irrelevant or later-positioned documents. Recent work proposes the differential attention mechanism to address this issue, but this mechanism is limited by an unsuitable common-mode rejection ratio (CMRR) and high computational costs. Inspired by the operational amplifier (OpAmp), we propose the OpAmp adaptation to address these challenges, which is implemented with adapters efficiently. By integrating the adapter into pre-trained Transformer blocks, our approach enhances focus on the golden context without costly training from scratch. Empirical evaluations on noisy-context benchmarks reveal that our Qwen2.5-OpAmp-72B model, trained with our OpAmp adaptation, surpasses the performance of state-of-the-art LLMs, including DeepSeek-V3 and GPT-4o.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。