通过编辑级归因提升语法纠错评估指标的可解释性。
Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction
- 用博弈论中的Shapley值计算每处修改对总分的贡献。
- 不同粒度下结果一致，与人工评价约70%吻合。
- 发现指标对拼写修改存在忽略倾向，适合模型分析与反馈优化。
语法纠错（GEC）已有多种评估指标，但许多参考无关指标缺乏可解释性，阻碍研究者分析模型优劣，也限制了向用户提供详细反馈。为此，本文提出将句子级评分归因于具体编辑操作，揭示各修正对整体表现的贡献。采用合作博弈论中的Shapley值计算每个编辑的贡献。在现有句子级指标上的实验表明，结果在不同编辑粒度下具高一致性，且与人工评价约70%对齐。进一步基于归因结果分析指标偏差，发现其倾向于忽略拼写类修改。代码已开源：https://github.com/naist-nlp/gec-attribute。
原文摘要 · Abstract (English)
Various evaluation metrics have been proposed for Grammatical Error Correction (GEC), but many, particularly reference-free metrics, lack explainability. This lack of explainability hinders researchers from analyzing the strengths and weaknesses of GEC models and limits the ability to provide detailed feedback for users. To address this issue, we propose attributing sentence-level scores to individual edits, providing insight into how specific corrections contribute to the overall performance. For the attribution method, we use Shapley values, from cooperative game theory, to compute the contribution of each edit. Experiments with existing sentence-level metrics demonstrate high consistency across different edit granularities and show approximately 70\% alignment with human evaluations. In addition, we analyze biases in the metrics based on the attribution results, revealing trends such as the tendency to ignore orthographic edits. Our implementation is available at \url{https://github.com/naist-nlp/gec-attribute}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。