让大模型推理更聚焦，避免关键步骤被忽略。
Attention-Aligned Reasoning for Large Language Models
- 利用推理结构引导注意力，防止重要步骤被淹没。
- 在六个基准上超越现有方法，最高提升15.39%。
- 非推理模型也能媲美甚至超过同规模推理模型。
大语言模型在解决复杂任务时常生成冗长的推理链，但随着链条变长，关键中间步骤和原始提示会被上下文淹没，导致注意力不足并引发错误。本文提出ATAR，一种基于模型内在推理结构来引导注意力的新方法。实验表明，ATAR在六个基准测试中均优于当前最优方法，最高实现15.39%的绝对性能提升。此外，使用ATAR后，'非推理'模型在多数基准上的表现可与同尺寸推理模型媲美，甚至更优。消融实验显示，注意力对齐组件贡献显著，且在不同注意力引导后端下结果依然稳定。
原文摘要 · Abstract (English)
Large Language Models (LLMs) tend to generate a long reasoning chain when solving complex tasks. However, as the reasoning chain extends, critical intermediate steps and the original prompt will be buried in the context, receiving insufficient attention and leading to errors. In this work, we present ATAR, a novel reasoning method that leverages the inherent reasoning structure to steer LLM attention. Our experiments show that ATAR outperforms SOTA methods across six benchmarks, achieving up to 15.39% absolute improvement. Furthermore, with ATAR, "non-reasoning" models achieve comparable or even better performance compared to reasoning models of the same size in most benchmarks. Finally, our ablation studies show that the attention alignment component contributes significantly, and that these improvements are persist under different attentionsteering backends.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。