成功条件化可精确求解信任区域优化问题，实现安全可控的策略改进。
Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success
- 通过动作影响度量，将成功轨迹的模仿转化为约束优化问题。
- 策略改进量与动作影响严格相等，确保不产生性能退化或分布偏移。
- 适用于需要安全策略更新的强化学习场景，尤其适合对鲁棒性要求高的任务。
一种广泛使用的策略改进方法是成功条件化：收集轨迹，识别达成目标的成功路径，并让策略模仿这些路径上的行为。该思想在不同名称下出现——拒绝采样结合监督微调（SFT）、目标条件化强化学习、决策变换器等——但其解决的优化问题始终不明。本文证明，成功条件化恰好求解一个信任区域优化问题，最大化策略改进，同时受χ²散度约束，其半径由数据自动决定。由此得到一个恒等式：相对策略改进、策略变化幅度，以及我们定义的“动作影响”（衡量动作随机性如何影响成功率）在每个状态上完全相等。因此，成功条件化自然成为一种保守的改进算子。精确的成功条件化不会降低性能，也不会引发危险的分布偏移；当它失效时，会明显表现为几乎不改变策略。我们将此理论应用于常见的回报阈值法，表明其能放大改进效果，但可能与真实目标产生偏差。
原文摘要 · Abstract (English)
A widely used technique for improving policies is success conditioning, in which one collects trajectories, identifies those that achieve a desired outcome, and updates the policy to imitate the actions taken along successful trajectories. This principle appears under many names -- rejection sampling with SFT, goal-conditioned RL, Decision Transformers -- yet what optimization problem it solves, if any, has remained unclear. We prove that success conditioning exactly solves a trust-region optimization problem, maximizing policy improvement subject to a $χ^2$ divergence constraint whose radius is determined automatically by the data. This yields an identity: relative policy improvement, the magnitude of policy change, and a quantity we call action-influence -- measuring how random variation in action choices affects success rates -- are exactly equal at every state. Success conditioning thus emerges as a conservative improvement operator. Exact success conditioning cannot degrade performance or induce dangerous distribution shift, but when it fails, it does so observably, by hardly changing the policy at all. We apply our theory to the common practice of return thresholding, showing this can amplify improvement, but at the cost of potential misalignment with the true objective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。