arXiv:2608.24214cs.AI2026-08

让AI更懂何时该继续查、何时该回答,提升问答准确率与效率。

MetaRAG: Belief-Action Aligned Policy Optimization for Agentic RAG

论文配图:MetaRAG: Belief-Action Aligned Policy Optimization for Agentic RAG
图 1 · 摘自论文原文
  • 通过验证先行生成和内部信念探测,让AI自检是否已有足够证据
  • 设计一致性奖励机制,避免错误但看似合理的回答被强化
  • 训练时引入小开销,推理无额外负担,适合各类模型和场景

智能检索增强生成(Agentic RAG)要求语言模型判断何时继续搜索、何时作答。现有基于强化学习的方法依赖外部监督,忽略模型自身对当前证据是否充分的内部信念。为此,本文提出MetaRAG,一种信念-动作对齐的策略优化框架。MetaRAG采用验证先行生成,在每次实际动作前触发显式验证过程;并通过内部信念探测,从同一问题-历史上下文中估计模型自身的可回答性信念。基于此,构建一致性奖励,并由答案正确性门控,防止强化内部一致但错误的推理路径。信念探测仅用于训练,不增加推理开销。在七个公开QA基准上的实验表明,MetaRAG在准确率-效率权衡上持续优于强基线,且效果可迁移至深度研究场景、不同优化器及多种模型主干。

原文摘要 · Abstract (English)

Agentic retrieval-augmented generation (RAG) requires language models to decide when to continue searching and when to answer. Existing RL-based methods rely on external supervision and overlook the agent's internal belief about whether the current evidence is sufficient. To address this problem, we reformulate the search decision quality as belief-action alignment and propose MetaRAG, a belief-action aligned policy optimization framework for agentic RAG. MetaRAG uses Verify-first Action Generation to elicit an explicit verification process before each actual action, and Internal Belief Probing to estimate the policy model's own answerability belief from the same question-history context. Based on these, MetaRAG derives a consistency reward that is further gated by answer correctness, avoiding reinforcement of internally consistent but incorrect trajectories. The belief probe is used only during training and introduces no inference-time overhead. Experiments on seven public QA benchmarks show that MetaRAG consistently improves the accuracy-efficiency trade-off over strong RL-based agentic RAG baselines, with gains that transfer to deep research settings, different optimizers, and multiple model backbones.

RAG强化学习智能体问答系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。