提出可恢复性判断机制，让强化学习模型更聪明地决定何时纠正错误。
Not Every Divergence Should Be Suppressed: Counterfactual Recoverability in On-Policy Distillation
- 通过反事实实验判断错误状态能否修复，指导训练策略选择。
- 在AIME2025上达到0.578成功率，优于最佳基线0.517。
- 适合需要高精度决策的强化学习场景，如复杂推理任务。
在线策略蒸馏（OPD）监督学生访问的轨迹，但基于发散性的规则无法判断错误前缀是否仍可修正。本文将此判断问题形式化为反事实可恢复性，并通过预算匹配的教师延续与回滚分支重现每个错误状态。根据其相对成功情况，状态被分类为可恢复、不可逆但可避免或模糊三类，这些标签决定训练是否保留、回滚或常规监督对应轨迹。在AIME分支诊断中，可恢复状态的延续减回滚效应均值为0.185，不可逆但可避免状态为-1.000，体现相反干预偏好。基于分支的可恢复性代理实现AUC 1.000，显著优于仅用发散性指标的0.392。在冻结评估中，基于可恢复性的控制策略达到最强记录性能，在未见的AIME2025上达成0.578成功率，优于最佳基线的0.517；同时将AIME2024-2025 average@32从0.2656提升至0.3125，GPQA-Diamond average@32从0.2702提升至0.3070。组件消融显示，保留教师可修正前缀贡献最大。这些发现确立可恢复性作为OPD中选择性监督的结果导向变量。
原文摘要 · Abstract (English)
On-policy distillation (OPD) supervises student-visited trajectories, yet divergence-based rules cannot determine whether an erroneous prefix remains correctable. We formulate this decision as counterfactual recoverability and replay each error state through budget-matched teacher-continuation and rollback branches. Based on their relative success, states are categorized as recoverable, irreversible-but-avoidable, or ambiguous, and these labels guide whether training retains, rolls back, or conventionally supervises the corresponding trajectory. On AIME branch diagnostics, the mean continuation-minus-rollback effect is 0.185 for recoverable states and -1.000 for irreversible-but-avoidable states, demonstrating opposite intervention preferences. A branch-derived recoverability proxy achieves an AUC of 1.000, substantially outperforming divergence alone at 0.392. Across frozen evaluations, recoverability-aware control achieves the strongest recorded performance, reaching 0.578 success on held-out AIME2025 compared with 0.517 for the best baseline. It also improves AIME2024-2025 average@32 from 0.2656 to 0.3125 and GPQA-Diamond average@32 from 0.2702 to 0.3070. Component ablations further show that retaining teacher-correctable prefixes provides the largest individual contribution. These findings establish recoverability as an outcome-grounded decision variable for selective supervision in OPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。