arXiv:2606.29713cs.CLcs.AI2026-06中稿 · ICML被引 1

让大模型自检事实归属,输出可审计的详细依据。

SEVA: Self-Evolving Verification Agent with Process Reward for Fact Attribution

论文配图:SEVA: Self-Evolving Verification Agent with Process Reward for Fact Attribution
图 1 · 摘自论文原文
  • 设计结构化验证代理,输出证据对齐与分步推理链。
  • 过程奖励机制使准确率从64.9提升至69.0,格式正确率达100%。
  • 支持自我演化,适合需要高可信度验证的应用场景。

幻觉是基于大模型代理的可靠性瓶颈,而事实归属验证器是最后一道防线——但现有验证器仅输出模糊的二值标签,导致代理无法自我修正,操作者也无法审计。我们提出SEVA,一种结构化验证代理,可输出证据对齐、逐步推理链、校准置信度及六类可操作的错误诊断。训练此类代理面临挑战:标准二值奖励在多组件输出上引发优势崩溃——组内奖励方差消失,导致GRPO梯度消失。我们通过过程奖励解决此问题,将验证质量分解为五个独立成分,70/30加权过程信号,恢复梯度并诱导隐式课程:代理先掌握验证行为(对齐度0.917→0.997,格式72%→100%),再优化结果(F1 64.9→69.0)。结构化输出还支持‘验证→反思→探测→优化’自演化循环,四轮迭代后,在7B模型上揭示出意外结构发现:每轮生成的是领域专用模型,而非通用模型(在HaluEval上+15个百分点,TruthfulQA上-10至-14个百分点,4倍数据下仍持续)。在ClearFacts上,SEVA-3B达到69.0 F1,接近GPT-4o-mini的69.8,同时输出更丰富、可审计。

原文摘要 · Abstract (English)

Hallucination is the reliability bottleneck for LLM-based agents, and fact attribution verifiers are the last line of defense -- yet today's verifiers emit only opaque binary labels, leaving agents unable to self-correct and operators unable to audit. We present SEVA, a structured verification agent that emits evidence alignments, step-by-step reasoning chains, calibrated confidence, and a six-category error diagnosis with actionable fixes. Training such an agent with RL is non-trivial: standard binary reward on multi-component output triggers advantage collapse -- within-group reward variance vanishes and the GRPO gradient disappears. We resolve this with a process reward that decomposes verification quality into five independent components weighted 70/30 toward process signals, restoring the gradient and inducing an implicit curriculum -- the agent first masters verification behavior (alignment 0.917 -> 0.997, format 72% -> 100%), then outcomes (F1 64.9 -> 69.0). Structured output further enables a Verify -> Reflect -> Probe -> Refine self-evolution loop, which over four rounds on a 7B model surfaces an unexpected structural finding: each round produces a benchmark-specialist, not a generalist (+15 pp on HaluEval, -10 to -14 pp on TruthfulQA in the same model, persistent at 4x data). On ClearFacts, SEVA-3B matches GPT-4o-mini (69.0 vs. 69.8 F1) while producing substantially richer, auditable output -- confirming a principle that should generalize: for any RL task with multi-component generation, reward granularity must match output granularity.

大模型验证自演化强化学习事实溯源

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。