arXiv:2605.08327cs.LGcs.AI2026-05被引 3

让AI写表单时能自我纠错,靠的是双人博弈训练。

Interactive Critique-Revision Training for Reliable Structured LLM Generation

论文配图:Interactive Critique-Revision Training for Reliable Structured LLM Generation
图 1 · 摘自论文原文
  • 用生成器和验证器对抗训练,生成后可被质疑修正。
  • 在税务数据集上准确率超零样本和单一强化学习方法。
  • 适合需要高可靠性的表格填写、合规检查等场景。

在表单填写、合规检查和维护报告等结构化决策流程中,大模型输出需局部正确、全局一致且可审计。现有优化方法依赖启发式辩论或模型自生成监督,引发二次可信性问题。本文提出DPA-GRPO(双配对动作组相对策略优化),一种两玩家生成-验证博弈的配对动作训练方法。生成器提出输出,可在被挑战时修订;验证器可保持沉默,或提出安全保证论据(SAC),包含主张、论证与证据。这些SAC/无SAC与保留/修订决策构成配对反事实动作组,用于角色特异性KL正则化GRPO更新。我们分析无正则博弈,发现对严格低收益干预或修订动作赋予正概率会引发有利的单边偏离。在标准随机近似假设下,DPA-GRPO跟踪对应博弈微分方程,其孤立渐近稳定极限点为角色局部最优下的平稳候选局部均衡。在TaxCalcBench TY24上的实验表明,DPA-GRPO在Qwen3-4B和Qwen3-8B模型上均优于零样本生成与仅生成器强化学习基线,提升正确静默接受率,减少漏检错误,并改善校准后的修订行为,体现生成器与验证器的共同提升。

原文摘要 · Abstract (English)

In structured decision-making workflows such as form filling, compliance checking, and maintenance reporting, LLM outputs must be locally correct, globally consistent, and auditable against task-specific rules. Existing refinement methods often rely on heuristic debate, self-play, or LLM-generated supervision, creating a second-order assurance problem. We propose DPA-GRPO (Dual Paired-Action Group-Relative Policy Optimization), a paired-action training method for a two-player generator--verifier game with structured verifier interventions. The generator proposes outputs and may revise them when challenged; the verifier either remains silent or raises a safety assurance case (SAC) containing a claim, argument, and evidence. These SAC/no-SAC and KEEP/REVISE decisions induce paired counterfactual action groups, which DPA-GRPO uses for role-specific KL-regularized GRPO updates. We analyze the unregularized game and show that positive probability on strictly lower-reward intervention or revision actions creates a profitable unilateral deviation. Under standard stochastic-approximation assumptions, DPA-GRPO tracks the corresponding game ODE, whose isolated asymptotically stable limit points are stationary and candidate local equilibria under role-wise local optimality. Experiments on TaxCalcBench TY24 show that DPA-GRPO improves structured decision accuracy over zero-shot generation and generator-only RL baselines across Qwen3-4B and Qwen3-8B. Training increases correct silent acceptance, reduces missed errors, and improves calibrated revision behavior, indicating gains for both generator and verifier.

结构化生成强化学习可靠性双人博弈

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。