现有知识编辑评测无法衡量编辑作用范围判断,效果几乎为零。
On Scope Classification and Current Knowledge-Editing Benchmarks: A Negative Result, with INLAY as a Gradient-Free Case Study
- 用冻结模型+外部记忆的INLAY系统获取每条查询的真实答案,验证路由决策效果。
- 1689次测试中,最优路由策略仅比静态策略高0.00分,无一例弃权获胜。
- 评测设计缺陷导致无法奖励拒绝错误编辑的能力,需重构基准测试体系。
基于记忆的知识编辑方法在SERAC系列中均依赖于作用范围判断:给定查询,存储的编辑是否适用?我们发现现有知识编辑评测完全无法衡量这一决策。通过构建无需梯度的INLAY编辑器(模型冻结,编辑存于外部可寻址内存,解码时沿单个词元的反嵌入方向添加偏置),我们在三个数据集、三种输入条件下对1689个查询执行所有候选路由动作。无论何种组合,由一个最优路由选择器决定最佳动作,其性能与固定的一行静态策略相差仅四舍五入至0.00分;任何查询级别的路由方法所能获得的最大提升为0.00分。弃权从未成为胜出动作。根本原因在于结构设计:这些是反事实评测,评估问题要求输出编辑后的答案,因此仅依赖参数化知识回答本质上即错误,且无负样本的评测无法奖励分类器拒绝对错的能力。该结论普遍适用于整个被评测的作用范围分类器家族。我们直接验证了机制:通过主动隐藏一半样本的自身编辑信息,使可用余地从精确的+0.0000提升至+0.0420,并首次赋予弃权以胜利机会。此外,我们还报告INLAY并非全能——WISE在Qwen2.5-7B CounterFact上表现更优,检索增强生成在严格匹配的RippleEdits上超越所有测试方法(包括INLAY)。自检中发现两个路由系统的漏洞,但未改变已发表结果的核心数值(仅在噪声范围内变动)。
原文摘要 · Abstract (English)
Every memory-based knowledge editor in the SERAC lineage depends on a scope decision: given a query, does a stored edit apply? We report that current knowledge-editing benchmarks cannot measure this decision at all. Using INLAY, a gradient-free editor we built to obtain exact per-query ground truth (the model is frozen, edits live in an external addressable memory, and applying an edit is a bias added along one token's unembedding direction at decode time), we execute every candidate router action on 1,689 queries spanning three datasets and three input conditions. An oracle router choosing the best action every time ties a one-line static policy to four decimal places in all nine dataset-by-condition cells: the maximum attainable gain of any per-query routing method is 0.00 points. Abstention is the sole winning action zero times out of 1,689. The cause is structural: these are counterfactual benchmarks whose evaluation question asks for the post-edit answer, so answering from parametric knowledge is wrong by construction, and a benchmark without negatives cannot reward a classifier's ability to reject. This generalizes beyond our system to the whole scope-classifier family the benchmarks are used to evaluate. We confirm the mechanism directly: constructing the missing condition ourselves, by withholding a query's own edit from the index for half the sample, moves pooled headroom from exactly +0.0000 to +0.0420 and gives abstention its first wins. We also report where INLAY itself does not win (WISE beats it on Qwen2.5-7B CounterFact, and retrieval-augmented generation beats every method we tested, INLAY included, on rigorously matched RippleEdits), and disclose two bugs found during a self-audit of our own routing machinery, neither of which changed a published headline number outside noise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。