用真实专家标注验证翻译模型,发现知识库回环让评估结果失真。
When the Knowledge Base Becomes the Gold Standard: Measuring Resource-Shared Evaluation Loops in Entity-Level Machine Translation
- 用独立于注入管道的专家标注作为真实标准
- 仅31.1%的人名不在知识库回环中,回环段准确率达97.8%
- 评估结果反映的是模型对知识库的依赖而非真实翻译能力
《世宗实录》是联合国教科文组织世界记忆名录文献,目前仅37.4%完成翻译。自动翻译中最显著的错误是人名误译——错误姓名会扭曲历史事实而非仅影响表面表达。低资源历史领域缺乏专家级金标准,研究者常以知识库(KB)替代。但该知识库正是系统注入的资源,导致评估自指:指标衡量的是指令遵循度而非翻译质量。本文通过国家韩史研究院提供的专家人名标注作为独立于注入流程的金标准,固定实体集,仅改变正确读音来源。在527个专家标注提及中,仅有31.1%位于注入管道之外。重叠段中,注入读音与人工翻译一致率达97.8%,而独立段仅为70.1%,表明看似健康的部分恰恰由回环维持。四模型差异分析显示,知识库注入带来的提升仅限于共享资源的段落,在独立段落中增益为零或负值。注入后保持率集中于0.910–0.996区间,尽管基线性能相差五倍,说明报告增益实为先前表现的补足,弱模型看似改善更明显。在移除构建过滤器的独立样本上,同一模型测量结果重合,不同模型间则不重合,证明该度量反映模型特性而非样本偏差。
原文摘要 · Abstract (English)
The Seungjeongwon Ilgi, a UNESCO Memory of the World record, is only 37.4% translated, and the most conspicuous failure mode in automatic translation is the person name -- a misread name corrupts the historical fact rather than merely the surface. Low-resource historical domains have no expert gold standard for entity translation, so practitioners substitute a knowledge base (KB) for the gold. That KB is the same resource injected into the system: scoring becomes self-referential and the metric measures instruction compliance rather than translation quality. We measure this loop. Using expert person-name annotations from the National Institute of Korean History as a gold independent of the injection pipeline, we hold the entity set fixed and vary only the provenance of the correct reading. Of 527 expert-annotated mentions, only 31.1% lie outside the injection pipeline, and the residual loop is not uniform -- in the overlapping segment the injected reading agrees with the human translation 97.8% of the time against 70.1% in the independent one, so the segment that looks healthiest is the one the loop is holding up. Across four models, a difference-in-differences analysis shows the gain from KB injection is confined to the segment whose gold shares the injected resource; in the independent segment it is at or below zero. Post-injection preservation clusters in a narrow 0.910-0.996 band even though baseline capability differs fivefold, so the reported gain is the complement of prior performance and weaker models appear to improve more dramatically. On an independent sample built by removing the construction filter, the measure replicates within model (overlapping intervals) while discriminating between models (non-overlapping intervals) -- it reflects a property of the model, not of the sample.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。