用行为等价性改进大模型智能体的强化学习优势估计，提升长序列任务成功率。
BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agents
- 基于动作反事实估计与双模拟聚类，重构优势计算的比较单元。
- 在ALFWorld上使成功率达97.1%，超越基线2.4个百分点且全种子达标。
- 无需额外训练或模型，仅增加11.3%计算开销，适合高效长程决策场景。
逐步分组强化学习是无需学习价值函数即可训练长时序大模型智能体的有效方法：它复用多条采样轨迹来估计局部优势。但其根本缺陷在于：每个组内相对估计器假设所比较的步骤在信用分配上是等价的。我们发现当前智能体变体存在状态-动作信用错配问题：观察哈希划分在状态侧过细，导致大量单例组（无步骤级信号），而动作侧单一组内均值又过粗，混淆了状态值与动作特异性信用。本文提出BiPACE（双模拟引导策略优化与动作反事实估计），一种无需额外评论家、辅助损失或新采样的即插即用优势估计器。BiGPO通过演员自身隐藏状态的余弦距离聚类步骤，以经验性策略诱导代理双模拟，显著降低观察哈希带来的单例率。PACE则在每类行为簇内使用动作条件的同行基准重新中心化回报；其Q式实例非参数化估计局部Q(s,a)−V(s)。在ALFWorld/Qwen2.5-7B上，BiPACE_Q将整体验证成功率从GiGPO的90.8提升至97.1±0.9（三种子），且所有种子均突破95%阈值（GiGPO未达）；在Qwen2.5-1.5B上达93.5±1.2，优于GiGPO的86.7；在WebShop和TextCraft上也优于GRPO与GiGPO。BiPACE特定开销仅为单步训练时间的11.3%。它将比较单位从表面一致性改为近似行为等价加动作侧反事实，实现本质改进。代码已公开于https://github.com/TianxiangZhao/BiPACE。
原文摘要 · Abstract (English)
Stepwise group-based RL is an attractive way to train long-horizon LLM agents without a learned critic: it reuses multiple sampled rollouts to estimate local advantages. Its weakness is less visible but more fundamental: every group-relative estimator assumes that the steps it compares are equivalent for credit assignment. We show that current agentic variants violate this assumption through a state-action credit mismatch. The observation-hash partition is overly fine on the state side, creating singleton groups with zero step-level signal, while a single within-group mean is too coarse on the action side, mixing state-value estimation with action-specific credit. We introduce BiPACE (Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation), a drop-in advantage estimator that fixes both sides without adding a critic, auxiliary loss, or extra rollouts. BiGPO clusters steps by cosine distance in the actor's own hidden-state geometry, an empirical policy-induced proxy for bisimulation that substantially lowers the singleton rate left by observation hashing. PACE then recenters returns within each behavioral cluster using action-conditioned peer baselines; its Q-style instance estimates a local Q(s,a)-V(s) nonparametrically. On ALFWorld/Qwen2.5-7B, BiPACE_Q raises overall validation success from GiGPO's 90.8 to $97.1\pm0.9$ over three seeds, and crosses the 95% threshold on every seed, which GiGPO never does within the same budget. On Qwen2.5-1.5B it reaches $93.5\pm1.2$ versus GiGPO's 86.7, and on WebShop and TextCraft it improves over GRPO and GiGPO at both model scales. The measured BiPACE-specific overhead is 11.3% of a single training-step wall time. Yet it changes the estimator's comparison unit from surface identity to approximate behavioral equivalence plus action-side counterfactuals. The code is available at https://github.com/TianxiangZhao/BiPACE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。