arXiv:2605.10488cs.CLcs.AI2026-05

用强化学习自动修复知识库缺陷,提升大模型任务表现

DeepRefine: Agentic Knowledge Refinement via Reinforcement Learning

论文配图:DeepRefine: Agentic Knowledge Refinement via Reinforcement Learning
图 1 · 摘自论文原文
  • 通过多轮交互诊断知识缺陷并精准修正
  • 在多个下游任务上显著超越基线模型
  • 适合需要高质量知识支持的智能代理场景

外部知识使大型语言模型代理能在开放、知识密集型任务中超越其参数化记忆进行决策。然而,知识库的质量常受限于不完整、错误或冗余,表现为证据缺失、文档间链接不足、置信度低或表述模糊等问题。这些缺陷在迭代使用中会累积,降低检索准确性和下游任务性能。本文提出 DeepRefine,一种基于强化学习的智能体知识精炼框架,可对预构建的结构化知识库(如知识图谱或 LLM-Wikis)进行动态优化,使其更适配下游任务。DeepRefine 与知识库进行多轮交互,基于交互历史执行溯因诊断,定位潜在缺陷,并执行针对性精炼操作以增量更新知识库。为在缺乏标准精炼轨迹的情况下优化精炼策略,引入增益超越草稿(GBD)奖励,并端到端训练推理过程。大量实验表明,DeepRefine 在多个下游任务中持续优于强基线。

原文摘要 · Abstract (English)

External knowledge enables large language model (LLM) agents to ground their actions and decisions beyond intrinsic parametric memory in open-ended, knowledge-intensive downstream tasks. Yet the quality of the underlying knowledge bases is systematically limited by incompleteness, incorrectness, or redundancy, manifested as missing evidence or cross-document links, low-confidence or imprecise claims, and ambiguous or coreference resolution issues. Such defects compound under iterative use, degrading retrieval fidelity and downstream task performance. We present \textbf{DeepRefine}, a reinforcement learning framework for agentic knowledge refinement that evolves the quality of any pre-constructed structured knowledge bases, e.g., knowledge graphs or LLM-Wikis, with user queries to make it more suitable for the downstream tasks. DeepRefine performs multi-turn interactions with the knowledge base and conducts abductive diagnosis over the interaction history, localizes likely defects, and executes targeted refinement actions for incremental knowledge base updates. To further optimize refinement policies of DeepRefine without golden refinement trajectories, we introduce a Gain-Beyond-Draft (GBD) reward and train the reasoning process end-to-end via reinforcement learning. Extensive experiments demonstrate consistent downstream gains over strong baselines.

知识精炼强化学习LLM代理知识图谱

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。