用编辑向量与最优传输提升语法纠错评估效果
Grammatical Error Correction Evaluation by Optimally Transporting Edit Representation
- 基于编辑操作构建向量表示，通过不平衡最优传输对齐假设与参考句
- 在SEEDA评测中显著提升性能，尤其在增加流畅性的场景下表现突出
- 运输方案可解释为软编辑对齐，适合系统排名与纠错分析
自动语法纠错（GEC）的评估对选择最佳系统至关重要。当前主流的参考依赖型指标通过衡量假设句与参考句之间的相似性进行评估，但基于嵌入的相似性度量（如BERTScore）常因源句中大量词汇在假设和参考中保持不变而失效。本研究聚焦专为GEC设计的编辑标注工具ERRANT，提出一种用于编辑的向量表示——编辑向量，并引入新度量UOT-ERRANT，利用不平衡最优传输将假设句的编辑向量映射到参考句。SEEDA元评估实验表明，UOT-ERRANT显著提升评估性能，特别是在+Fluency领域（编辑较多时）表现更优。该方法高度可解释，运输计划可视为软编辑对齐，适用于系统排序与纠错系统分析。代码已开源：https://github.com/gotutiyan/uot-errant。
原文摘要 · Abstract (English)
Automatic evaluation in grammatical error correction (GEC) is crucial for selecting the best-performing systems. Currently, reference-based metrics are a popular choice, which basically measure the similarity between hypothesis and reference sentences. However, similarity measures based on embeddings, such as BERTScore, are often ineffective, since many words in the source sentences remain unchanged in both the hypothesis and the reference. This study focuses on edits specifically designed for GEC, i.e., ERRANT, and computes similarity measured over the edits from the source sentence. To this end, we propose edit vector, a representation for an edit, and introduce a new metric, UOT-ERRANT, which transports these edit vectors from hypothesis to reference using unbalanced optimal transport. Experiments with SEEDA meta-evaluation show that UOT-ERRANT improves evaluation performance, particularly in the +Fluency domain where many edits occur. Moreover, our method is highly interpretable because the transport plan can be interpreted as a soft edit alignment, making UOT-ERRANT a useful metric for both system ranking and analyzing GEC systems. Our code is available from https://github.com/gotutiyan/uot-errant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。