让大模型推理具备自修复能力,解决早期错误导致全程崩溃的问题
Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMs

- 用内在认知不确定性动态调整奖励,实现局部逻辑错误精准清除
- 在DeepMath-103k上训练,4B/8B模型在AIME上分别提升5.349%/6.514%
- 无需外部信号,复用历史缓存,保持线性内存开销
尽管强化学习扩展了大语言模型的认知边界,但在长序列逻辑推理中仍受自回归诅咒困扰:生成初期的微小认知扰动会沿马尔可夫决策过程不可逆传播,引发级联失败导致推理轨迹崩溃。为克服这一问题,我们提出动态认知熵协同可擦除强化学习(E³RL)。E³RL不依赖外部信号,以模型内部自回归交叉熵作为认知不确定性的内在坐标,通过引入分段自适应动态阈值与优势分配机制,使模型能精准剔除局部逻辑缺陷,并复用历史键值缓存流,赋予推理过程自愈能力。我们在DeepMath-103k数据集上训练E³RL。实验表明,该方法显著提升长序列推理探索效率,提高样本效率的同时维持线性内存开销。在数学推理基准AIME上,4B与8B参数模型分别超越此前最先进水平5.349%和6.514%。这些结果表明,E³RL打破了长序列推理中的自回归诅咒,为下一代自愈型通用人工智能奠定了理论与系统基础。
原文摘要 · Abstract (English)
Although reinforcement learning (RL) has expanded the cognitive boundaries of large language models (LLMs), it often remains vulnerable to the autoregressive curse in long-horizon logical reasoning: small epistemic perturbations introduced early in generation can propagate irreversibly along the Markov decision process flow, triggering cascading failures that drive the reasoning trajectory toward collapse. To overcome this autoregressive cascade, in which a single early mistake can compromise all subsequent reasoning steps, we propose dynamic epistemic entropy orchestrated erasable reinforcement learning ($\text{E}^3\text{RL}$). $\text{E}^3\text{RL}$ eliminates reliance on external signals by grounding the model's endogenous local autoregressive cross-entropy as an intrinsic coordinate of epistemic uncertainty. By introducing segment-level adaptive dynamic thresholds and advantage allocation, $\text{E}^3\text{RL}$ enables the model to precisely excise localized logical defects while reusing historical key-value (KV) cache streams, thereby endowing the reasoning process with a self-healing capability. We train $\text{E}^3\text{RL}$ on the DeepMath-103k dataset. Experimental results show that $\text{E}^3\text{RL}$ reshapes the exploration efficiency of long-sequence reasoning and improves sample efficiency while maintaining linear memory overhead. On mathematical reasoning benchmarks such as AIME, $\text{E}^3\text{RL}$ achieves substantial performance gains, with the 4B and 8B parameter models surpassing previous state-of-the-art (SOTA) results by 5.349\% and 6.514\%, respectively. These findings suggest that $\text{E}^3\text{RL}$ shatters the autoregressive curse in long-sequence reasoning and establishes a theoretical and systems-level foundation for the next generation of self-healing artificial general intelligence (AGI).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。