arXiv:2607.04576cs.CLcs.CY2026-07

测试大模型维护维基时渐进披露的效率,发现直接定位页面反而更省资源。

Progressive Disclosure for LLM-Maintained Wiki Knowledge Bases: a Preregistered Ablation

  • 用渐进披露结构(仅加载摘要)替代完整索引,控制变量对比访问方式
  • 质量与全量索引相当,成本降低超一半,主要因减少工具调用和引用页数
  • 适合关注大模型知识管理效率与评估严谨性的研究者

LLM代理越来越多地在它们协助维护的知识库上回答问题。一种常见直觉认为,渐进披露(仅保留紧凑目录和每页一行摘要)可减少加载内容,使检索比使用大型单一索引更高效。我们在一个由LLM维护的709页Markdown维基上测试这一假设。对维基进行改造以支持渐进披露,并开展预注册消融实验:四个版本的语料库仅在代理获取内容的方式上不同,页面主体字节完全一致,冻结为不可变的git标签,因此所有差异仅来自访问结构。我们交叉了三种访问条件(协议约束代理、自由自路由代理、目录预加载),并由跨家族裁判盲评答案质量,参照已验证的黄金标准。初步结果颠覆了原有假设:一个能力强的工具使用代理从不加载索引,而是从问题中推断页面路径并直接读取,因此原本预期的节省并未实现。于是我们将答案质量设为首要目标,成本为次要。结果显示,质量与索引基线无显著差异(在预注册容差范围内),而成本在所有条件下均下降:自路由代理降低成本约三分之二,目录预加载下降幅超过一半,所有置信区间均不包含零。节省并非来自避免索引加载,而是源于更精准的访问——检索臂引用的页面更少,工具调用次数更少。该研究同时作为评估有效性的案例研究,采用威胁有效性分析方法来审视其自身工具链。

原文摘要 · Abstract (English)

LLM agents increasingly answer questions against knowledge bases they help maintain. A common intuition holds that progressive disclosure, a compact catalog plus a one-line summary per page so the agent loads only what it needs, should make this cheaper than consulting a large monolithic index. We test that on a real 709-page markdown wiki maintained by an LLM. We retrofit it for progressive disclosure and run a preregistered ablation in which four versions of the corpus differ only in how the agent reaches the content: page bodies are byte-identical across arms, frozen as immutable git tags, so any measured difference is due to access structure alone. We cross the arms with three access conditions (a protocol-constrained agent, a free self-routing agent, and a catalog-preload regime) and grade answers blind against verified gold references with a cross-family judge. A pilot upended the premise: a capable tool-using agent never loads the index, inferring a page's path from the question and reading it directly, so the specific saving the retrofit targets does not materialize. We therefore made answer quality primary and cost secondary. Quality is non-inferior (the retrieval arm matches the index baseline within the preregistered margin) while cost falls in every regime, from about a third for a self-routing agent to well over half under catalog-preload, all confidence intervals excluding zero. The saving comes not from avoiding the index load but from more targeted access: the retrieval arm cites fewer pages and takes fewer tool turns. The study doubles as a case study in evaluation validity, applying threat-to-validity discipline to the tooling that produced it.

大模型知识库效率优化评估方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。