arXiv:2604.08905cs.AIcs.LG2026-04被引 6

让大模型推理更稳:引入两种新指标提升逻辑一致性

StaRPO: Stability-Augmented Reinforcement Policy Optimization

  • 用自相关函数和路径效率衡量推理过程的局部连贯性和全局目标性
  • 在四个推理基准上同时提升答案准确率与逻辑稳定性,效果优于基线
  • 适合关注模型推理可信度、避免胡编乱造的研究者使用

强化学习在提升大语言模型复杂推理任务准确性方面表现优异。现有强化学习策略优化框架多依赖最终答案正确性作为反馈信号,很少捕捉推理过程的内部逻辑结构,导致模型生成看似流畅但逻辑不一致、结构混乱或冗余的回答。为此,我们提出StaRPO——一种增强稳定性的强化学习策略优化框架,将推理稳定性显式纳入优化目标。StaRPO将稳定性分解为两个可计算的轻量级指标:自相关函数(ACF)用于评估步骤间的局部连贯性,路径效率(PE)用于评估推理轨迹的全局目标导向性。这些稳定性奖励与任务奖励结合,提供互补且过程感知的反馈。我们在两个基础模型上验证了ACF与PE奖励与逻辑错误的相关性。在四个推理基准上的实验表明,StaRPO持续优于对比基线,能够同时提升最终答案准确率与逻辑稳定性。

原文摘要 · Abstract (English)

Reinforcement learning (RL) is effective in enhancing the accuracy of large language models in complex reasoning tasks. Existing RL policy optimization frameworks rely on final-answer correctness as feedback signals and rarely capture the internal logical structure of the reasoning process. Consequently, the models would generate fluent and semantically relevant responses but logically inconsistent, structurally erratic, or redundant. To this end, we propose StaRPO, a stability-augmented reinforcement learning framework that explicitly incorporates reasoning stability into the optimization objective. Our StaRPO decomposes stability into two computable lightweight metrics: the Autocorrelation Function (ACF) to evaluate local step-to-step coherence, and Path Efficiency (PE) to evaluate global goal-directedness of the reasoning trajectory. These stability rewards are combined with task rewards to provide complementary and process-aware feedback. We validate the effectiveness of using ACF and PE rewards by showing their correlation with logic errors on two backbone models. Experiments on four reasoning benchmarks show that StaRPO consistently outperforms compared baselines and can enhance both final-answer accuracy and logical stability.

强化学习逻辑推理大模型稳定性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。