模型自动生成可验证问题并迭代优化，无需人工标注答案。
ANCORA: Learning to Question via Manifold-Anchored Self-Play for Verifiable Reasoning
- 通过自我生成问题与求解的循环机制，实现无监督推理能力提升。
- 在测试时训练中将通过率从26.6%提升至81.5%，超越现有方法15.8个百分点。
- 适合追求自动推理与零样本迁移能力的研究者和开发者。
我们提出一种开放式课程自对弈范式：不依赖固定提示集进行回答学习，而是让统一策略学会提问——生成可验证问题、求解并利用验证器反馈实现自我改进，无需人工标注答案。引入ANCORA，其策略在生成器（Proposer）与求解器（Solver）间交替，依托三大核心机制：两级组相对更新，关联不同问题间的生成优势与不同尝试间的求解优势；迭代自蒸馏指令微调（SFT），将基础模型投影至有效输出流形；以及基于UCB引导的课程有向无环图（Curriculum DAG），使策略生成的问题集可证明在自组合下持续扩展。无这些稳定机制时，稀疏验证反馈导致生成器崩溃，即便采用MLRL对齐奖励亦然；而加入后，ANCORA从零人类解决方案启动可验证课程。在Verus中实例化后，ANCORA将Dafny2Verus pass@1从26.6%的SFT基线提升至81.5%（测试时训练，0样本），优于PSV自对弈15.8个百分点，且后者具有一次推理优势；在迁移设置中，使用Dafny2Verus种子训练，在未见的MBPP与HumanEval上分别达到36.2%和17.2% pass@1。
原文摘要 · Abstract (English)
We propose a paradigm shift toward open-ended curriculum self-play: rather than learning to answer on a fixed prompt set, a unified policy learns to question: generating verifiable problems, solving them, and turning verifier feedback into self-improvement without human-annotated solutions. We introduce ANCORA, in which the policy alternates between a Proposer that synthesizes novel specifications and a Solver that produces verified solutions, anchored by three load-bearing mechanisms: a two-level group-relative update coupling Proposer advantages across specifications with Solver advantages across solution attempts; iterative self-distilled SFT projecting the base model onto its valid-output manifold before RL; and a UCB-guided Curriculum DAG whose policy-induced problem set can provably expand under self-composition. Without these stabilizers, sparse verifier feedback drives Proposer collapse even under MLRL-aligned rewards; with them, ANCORA bootstraps a verifiable curriculum from zero human solutions. Instantiated in Verus, ANCORA lifts Dafny2Verus pass@1 from a 26.6% SFT baseline to 81.5% in test-time training (TTT, 0-shot), outperforming PSV self-play by 15.8 points despite PSV's 1-shot inference; in a transfer setting, training from Dafny2Verus seeds yields 36.2% and 17.2% pass@1 on held-out MBPP and HumanEval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。