通过截断偏差过大的推理轨迹，提升大模型主动推理的稳定性和效率
Reducing Belief Deviation in Reinforcement Learning for Active Reasoning
- 检测并截断导致信念偏离的无效推理尾部
- 在5个任务中提升性能最高达30点，减少34%的令牌消耗
- 适合构建稳定可靠的大模型主动推理系统
主动推理要求大语言模型（LLM）代理与外部源交互，在多轮对话中策略性地获取信息以解决问题。其核心是信念追踪：准确表征问题状态及其不确定性。然而，由于推理能力有限，基于LLM的代理常出现信念偏离：内部信念偏离真实状态，导致状态意识丧失和无意义或重复行为。一旦发生，错误在强化学习（RL）轨迹中累积，造成信用归因错误和探索受限。为此，我们提出跟踪信念偏差，并开发了$oldsymbol{T^3}$——一种简单而严谨的方法，可检测过度偏差并截断训练轨迹，抑制无效尾部影响。因此，$oldsymbol{T^3}$保留了有效前缀的信用，系统性提升策略优化。在5个挑战性任务上，$oldsymbol{T^3}$一致增强训练稳定性，性能最高提升30点，令牌成本降低最多34%。结果表明，信念控制是构建具备主动推理能力的鲁棒LLM代理的关键原则。
原文摘要 · Abstract (English)
Active reasoning requires large language model (LLM) agents to interact with external sources and strategically gather information to solve problems in multiple turns. Central to this process is belief tracking: maintaining an accurate representation of the underlying state and uncertainty in understanding and solving the problem. However, due to limited reasoning capabilities, LLM-based agents often suffer belief deviation: their internal beliefs drift from the true problem state, leading to loss of state awareness and uninformative or repetitive actions. Once this happens, errors compound in the trajectories used for reinforcement learning (RL), leading to misattributed credits and limited exploration. To address this issue, we propose to track belief deviation and develop $\mathbf{T^3}$, a simple yet principled method that detects excessive deviation and truncates training trajectories to suppress uninformative tail effects. Hence, $\mathbf{T^3}$ preserves credits for informative prefixes and systematically improves policy optimization. Across 5 challenging tasks, $\mathbf{T^3}$ consistently enhances training stability and yields performance gains of up to 30 points while cutting token cost by up to 34%. These results highlight belief control as a key principle for building robust LLM agents capable of active reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。