arXiv:2512.06869cs.CL2025-12

解决对话模型越聊越跑偏的问题,让大模型记住关键指令并精准调用回忆。

Rhea: Role-aware Heuristic Episodic Attention for Conversational LLMs

  • 将对话记忆拆分为指令库和情景库,分别管理全局规则和实时互动。
  • 在10分制评测中提升准确率1.04分,长对话下指令遵循率超8.1。
  • 适合需要高一致性对话的场景,如客服、教育助手等应用。

大型语言模型在单轮任务中表现优异,但在多轮对话中性能逐渐下降。我们将其归因于累积上下文衰减——由注意力污染、稀释和漂移导致的上下文完整性退化。为此,我们提出Rhea(角色感知启发式情景注意力)框架,将对话历史解耦为两个功能独立的记忆模块:(1) 指令记忆(IM),通过结构优先机制持久存储高保真全局约束;(2) 情景记忆(EM),通过非对称噪声控制与启发式上下文检索动态管理用户-模型交互。推理时,Rhea通过优先注意力机制构建高信噪比上下文,选择性融合相关情景信息,同时始终优先保留全局指令。在MT-Eval和Long-MT-Bench+等多个多轮对话基准测试中,Rhea有效缓解性能衰退,在10分制评分中较强基线提升1.04分(相对增益16%)。此外,其在长时交互中保持近似完美的指令遵循度(IAR > 8.1)。结果表明,Rhea为构建更精确、指令一致的对话型LLM提供了系统性解决方案。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have achieved remarkable performance on single-turn tasks, yet their effectiveness deteriorates in multi-turn conversations. We define this phenomenon as cumulative contextual decay - a progressive degradation of contextual integrity caused by attention pollution, dilution, and drift. To address this challenge, we propose Rhea (Role-aware Heuristic Episodic Attention), a novel framework that decouples conversation history into two functionally independent memory modules: (1) an Instructional Memory (IM) that persistently stores high-fidelity global constraints via a structural priority mechanism, and (2) an Episodic Memory (EM) that dynamically manages user-model interactions via asymmetric noise control and heuristic context retrieval. During inference, Rhea constructs a high signal-to-noise context by applying its priority attention: selectively integrating relevant episodic information while always prioritizing global instructions. To validate this approach, experiments on multiple multi-turn conversation benchmarks - including MT-Eval and Long-MT-Bench+ - show that Rhea mitigates performance decay and improves overall accuracy by 1.04 points on a 10-point scale (a 16% relative gain over strong baselines). Moreover, Rhea maintains near-perfect instruction fidelity (IAR > 8.1) across long-horizon interactions. These results demonstrate that Rhea provides a principled and effective framework for building more precise, instruction-consistent conversational LLMs.

对话模型注意力机制指令遵循

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。