让智能体聚焦关键决策点，提升长程任务效率与效果
CARL: Criticality-Aware Agentic Reinforcement Learning
- 用熵衡量状态重要性，只优化关键步骤的策略
- 在多个任务上性能超越传统方法，训练速度更快
- 适合需要长期规划与高效推理的复杂任务场景
能够通过与环境多次交互完成复杂任务的智能体已成为研究热点。然而，在多步设置中，传统群体级策略优化算法因假设每一步贡献均等而表现不佳，这与实际严重不符。我们分析发现，仅少数状态上的动作选择对最终结果起决定性作用。基于此，提出面向长时程代理推理的临界感知强化学习算法CARL。CARL利用熵作为状态临界性的启发式代理，通过仅对高临界性状态采取的动作赋予奖励，并将低临界性状态的动作排除在模型更新之外，从而避免噪声信用分配和冗余计算。大量实验表明，CARL在多种评估设置下均实现更强性能与更高效率。源代码将公开。
原文摘要 · Abstract (English)
Agents capable of accomplishing complex tasks through multiple interactions with the environment have emerged as a popular research direction. However, in such multi-step settings, the conventional group-level policy optimization algorithm becomes suboptimal because of its underlying assumption that each step holds equal contribution, which deviates significantly from reality. Our analysis reveals that only the action choices on a small fraction of states are critical in determining the final outcome. Building on this insight, we propose CARL, a criticality-aware reinforcement learning algorithm tailored for long-horizon agentic reasoning. CARL leverages entropy as a heuristic proxy for state criticality and achieves focused training by assigning rewards to actions taken from high-criticality states while excluding actions taken from low-criticality states from model updates, avoiding noisy credit assignment and redundant computation. Extensive experiments demonstrate that CARL achieves both stronger performance and higher efficiency across diverse evaluation settings. The source code will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。