让智能体通过问答反馈动态优化知识库结构,提升检索准确率。
Training a Knowledge Base: Supervised Structure Learning for Agent-Curated Document Stores

- 用问答对监督训练知识库,智能体根据答案实时修改文档结构。
- 在训练问题上达0.294 F1,比基线少25%操作,准确率提升2.1倍。
- 适合需要高效更新知识库的对话系统或智能助手开发者。
传统检索增强生成将文档库视为静态输入,现有构建结构的方法多为无监督,全库统一索引,缺乏对问题所需结构的信号。本文将知识库视为非参数化模型,通过(问题,答案)对进行监督训练:由智能体代理回答当前库中的问题,获知正确答案后编辑知识库。知识库持续演化,测试使用两个无污染基准:新发布的虚构世界生成器KBGym和PhantomWiki。通过四类问题组评估泛化能力——训练问题本身、共享双键、单键或无键的问题。结果表明,随着问题与训练集重叠度提高,优势逐渐显现:无键时性能持平,双键时提升0.176 F1,训练问题上达+0.294 F1且减少25%操作。相较HippoRAG仅用1,913条链接即实现其196,112条链接的精度增益,单位覆盖范围下行动节省1.5倍,准确率提升2.1倍。知识库覆盖率越高,准确率越稳定提升,扩大训练问题数量可同步提升覆盖与泛化能力。
原文摘要 · Abstract (English)
Retrieval-augmented generation treats the document store as a frozen input, and the offline pipelines that do build structure over it build it unsupervised -- a whole corpus indexed at uniform effort, with no signal about which structure a question will need. We instead treat the knowledge base as a non-parametric model trained on (question, answer) pairs: a curator agent answers a supervised question against the current store, is shown the gold answer, then edits the store. The store carries forward, and we evaluate the curated store with a test set, on two contamination-free benchmarks: KBGym, a fictional-universe generator we release, and PhantomWiki. Generalization is probed with four question groups of decreasing overlap with the training set: the trained questions themselves, and unseen questions sharing both of their keys with training, one key, or neither. The curated store's advantage grows with overlap -- from parity where no key was shared, through +0.176 F1 where both keys were, to 25% fewer actions at +0.294 F1 on the trained questions, the one cell significant on both benchmarks -- while matching HippoRAG's gains with 1,913 links against its 196,112: per point of corpus covered, 1.5x the action saving and 2.1x the accuracy gain. Accuracy rises steadily with the share of the corpus the indexes cover, so training on more questions widens coverage, and with it the generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。