让AI模型学会在错误推理中自我修复,提升可靠性。
Learning Robust Reasoning through Guided Adversarial Self-Play
- 模型内部自动生成对抗性错误,训练自我诊断与纠错能力。
- 在4个1.5B~8B模型上,修复成功率显著提升,干净输入准确率也提高。
- 无需人工标注,适合打造鲁棒性强的推理系统。
基于可验证奖励的强化学习(RLVR)能生成强大的推理模型,但在条件信息有误时可能灾难性失效(如思维链被污染、部分解法误导或输入轻微扰动),因为标准方法仅在干净条件下优化最终答案正确性。本文提出GASP(引导式对抗自对弈)方法,通过仅依赖结果验证来显式训练检测与修复能力。在单一模型内构建对抗自对弈机制:污染者学习以局部一致的方式制造错误,而代理则在相同污染条件下学习诊断并恢复。为解决训练初期修复成功案例稀缺的问题,提出分布内修复指导,即对自生成修复进行模仿学习,提升恢复概率同时保留已有能力。在四个开源模型(1.5B–8B)上,GASP将原本脆弱的强推理模型转变为能抵御误导和扰动的鲁棒模型,且常提升干净条件下的准确率。进一步分析表明,对抗性污染形成有效课程,分布内指导使修复学习快速展开且表征漂移极小。
原文摘要 · Abstract (English)
Reinforcement learning from verifiable rewards (RLVR) produces strong reasoning models, yet they can fail catastrophically when the conditioning context is fallible (e.g., corrupted chain-of-thought, misleading partial solutions, or mild input perturbations), since standard RLVR optimizes final-answer correctness only under clean conditioning. We introduce GASP (Guided Adversarial Self-Play), a robustification method that explicitly trains detect-and-repair capabilities using only outcome verification. Without human labels or external teachers, GASP forms an adversarial self-play game within a single model: a polluter learns to induce failure via locally coherent corruptions, while an agent learns to diagnose and recover under the same corrupted conditioning. To address the scarcity of successful recoveries early in training, we propose in-distribution repair guidance, an imitation term on self-generated repairs that increases recovery probability while preserving previously acquired capabilities. Across four open-weight models (1.5B--8B), GASP transforms strong-but-brittle reasoners into robust ones that withstand misleading and perturbed context while often improving clean accuracy. Further analysis shows that adversarial corruptions induce an effective curriculum, and in-distribution guidance enables rapid recovery learning with minimal representational drift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。