解决大模型推理时重复旧思路的问题,提升真实反思能力。
What Makes Reasoning Invalid: Echo Reflection Mitigation for Large Language Models
- 通过信息过滤阻止错误中间结果影响最终答案
- 动态调节探索与利用,促进多样化反思
- 适用于需要深度思考的复杂任务场景
大语言模型在各类推理任务中表现优异,尤其在数学推理方面取得进展。然而在涉及领域专业知识的复杂任务中,模型在反思阶段常出现机械重复前期推理步骤的现象,即‘回声式反思’。我们发现其根源在于:(1)响应生成过程中信息流不可控,早期错误认知未经抑制传播至最终决策;(2)反思阶段内部知识探索不足,导致反复输出已有结论而非新见解。为此提出自适应熵策略优化(AEPO)方法,包含两个核心组件:(1)反思感知的信息过滤机制,量化认知信息流并阻断有害中间信息对最终答案的影响;(2)自适应熵优化,动态平衡各推理阶段的探索与利用,兼顾反思多样性与答案正确性。大量实验表明,AEPO在多个基准测试中均显著优于主流强化学习基线,实现最先进性能。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of reasoning tasks. Recent methods have further improved LLM performance in complex mathematical reasoning. However, when extending these methods beyond the domain of mathematical reasoning to tasks involving complex domain-specific knowledge, we observe a consistent failure of LLMs to generate novel insights during the reflection stage. Instead of conducting genuine cognitive refinement, the model tends to mechanically reiterate earlier reasoning steps without introducing new information or perspectives, a phenomenon referred to as "Echo Reflection". We attribute this behavior to two key defects: (1) Uncontrollable information flow during response generation, which allows premature intermediate thoughts to propagate unchecked and distort final decisions; (2) Insufficient exploration of internal knowledge during reflection, leading to repeating earlier findings rather than generating new cognitive insights. Building on these findings, we proposed a novel reinforcement learning method termed Adaptive Entropy Policy Optimization (AEPO). Specifically, the AEPO framework consists of two major components: (1) Reflection-aware Information Filtration, which quantifies the cognitive information flow and prevents the final answer from being affected by earlier bad cognitive information; (2) Adaptive-Entropy Optimization, which dynamically balances exploration and exploitation across different reasoning stages, promoting both reflective diversity and answer correctness. Extensive experiments demonstrate that AEPO consistently achieves state-of-the-art performance over mainstream reinforcement learning baselines across diverse benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。