提出精确评估文档修改的切除评分,解决传统方法被共用内容干扰的问题。
Excision Score: Evaluating Edits with Surgical Precision
- 基于移除共用内容后比对差异部分,模拟手术清场思路
- 在HumanEvalFix上比SARI高12%相关性,比BLEU高30%以上
- 适合代码编辑等需精准衡量改动的场景,尤其对共享上下文敏感
许多任务涉及文档(代码或文本)的编辑。本文将修订相似性问题形式化,统一多种机器学习评估任务的目标:评估对现有文档的修改。观察发现,修订通常只改变文档的一小部分,因此原文档与其修订版本共享大部分内容。为此提出五项修订相似性度量的合理性标准,旨在与人类判断对齐。结果显示,主流的成对度量如BLEU因受共享内容主导而失效——当人类认为差异明显时,它们仍报告高相似性。本文提出新型静态度量Excision Score(ES),通过计算最长公共子序列(LCS)先剔除原文档与真实/预测修订中共享的内容,仅比较剩余差异区域。这类似于外科医生创建无菌区聚焦操作区域。采用近似算法将标准立方复杂度的LCS加速至二次方。在代码编辑评估中,静态度量常作为测试通过率的低成本代理,实验表明ES优于现有度量。在HumanEvalFix上,其与测试执行的皮尔逊相关性比最接近的对手SARI高出12%,比标准度量如BLEU高出超过21%。关键标准为对共享上下文的不变性:当增加HumanEvalFix中的共享上下文扰动时,ES相对于SARI的提升增至20%,相对于标准度量提升超过30%。此外,ES还能处理其他度量无法应对的边界情况,如正确对齐移动的代码块,并适当奖励匹配的插入或删除。
原文摘要 · Abstract (English)
Many tasks revolve around editing a document, whether code or text. We formulate the revision similarity problem to unify a wide range of machine learning evaluation problems whose goal is to assess a revision to an existing document. We observe that revisions usually change only a small portion of an existing document, so the existing document and its immediate revisions share a majority of their content. We formulate five adequacy criteria for revision similarity measures, designed to align them with human judgement. We show that popular pairwise measures, like BLEU, fail to meet these criteria, because their scores are dominated by the shared content. They report high similarity between two revisions when humans would assess them as quite different. This is a fundamental flaw we address. We propose a novel static measure, Excision Score (ES), which computes longest common subsequence (LCS) to remove content shared by an existing document with the ground truth and predicted revisions, before comparing only the remaining divergent regions. This is analogous to a surgeon creating a sterile field to focus on the work area. We use approximation to speed the standard cubic LCS computation to quadratic. In code-editing evaluation, where static measures are often used as a cheap proxy for passing tests, we demonstrate that ES surpasses existing measures. When aligned with test execution on HumanEvalFix, ES improves over its nearest competitor, SARI, by 12% Pearson correlation and by >21% over standard measures like BLEU. The key criterion is invariance to shared context; when we perturb HumanEvalFix with increased shared context, ES' improvement over SARI increases to 20% and >30% over standard measures. ES also handles other corner cases that other measures do not, such as correctly aligning moved code blocks, and appropriately rewarding matching insertions or deletions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。