arXiv:2605.11636cs.AI2026-05

让大模型在干扰中自我进化,提升推理鲁棒性。

Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning

论文配图:Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning
图 1 · 摘自论文原文
  • 用对抗自博弈生成干扰上下文,逼模型识别核心逻辑。
  • 7个数学基准上平均提升7.2~10.2分,30B模型效果最佳。
  • 能暴露闭源模型缺陷,适合研究抗干扰推理的学者。

我们提出Seirênes,一种基于可验证奖励的强化学习框架,将大模型推理中的上下文干扰从失败模式转化为内部训练信号,实现更稳健推理模型的协同进化。尽管已有强化学习显著提升了推理能力,但模型在非理想化场景下仍显脆弱:如存在冗余信息、无关指令或偶然关联等,与标准基准的纯净分布不符。Seirênes通过参数共享的对抗自博弈循环,使单一模型同时承担构建迷惑性上下文和从中提取核心任务逻辑的双重目标。在持续对抗中,模型被强制摆脱表面模式匹配,转向深层逻辑推理。该机制形成动态进化的训练课程。在7个数学推理基准及4B至30B规模的模型上,平均性能提升分别为+10.2、+9.1和+7.2分。此外,4B Seirênes生成的干扰上下文使顶级闭源模型(GPT与Gemini)准确率下降约4–5分,揭示其发现推理模型盲点的通用能力。

原文摘要 · Abstract (English)

We present Seirênes, a self-play RL framework that transforms contextual interference from a failure mode of LLM reasoning into an internal training signal for co-evolving more resilient reasoners. While RL with verifiable rewards has significantly advanced reasoning capabilities, models can still exhibit fragility when encountering non-idealized contexts: scenarios characterized by superfluous information, tangential instructions, or incidental correlations that differ from the clean distributions typical of standard benchmarks. Seirênes harnesses this vulnerability through a parameter-shared and adversarial self-play loop. Within this framework, a single model is trained to both construct plausible yet distracting contexts that expose its own reasoning blind spots, and solve problems by discerning the essential task from these perturbations to recover the core underlying logic. By pitting these competing objectives against each other, Seirênes compels the model to move beyond superficial pattern matching and anchors its capabilities in robust underlying reasoning. This continuous interaction sustains an informative co-evolutionary curriculum as the model improves. Across seven mathematical reasoning benchmarks and model scales from 4B to 30B, Seirênes achieves average gains of +10.2, +9.1, and +7.2 points. Besides, distracting contexts produced by the 4B Seirênes model reduce the accuracy of top-tier closed-source models (GPT and Gemini) by roughly 4--5 points, revealing Seirênes' general ability to uncover reasoning models' blind spots.

大模型推理自博弈鲁棒性强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。