分析注意力头如何影响翻译中代词消歧，发现部分头可提升准确率。
Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models
- 通过观察和修改注意力分数，研究代词消歧中注意力头的作用。
- 优化关键注意力头后，代词消歧准确率最高提升5个百分点。
- 适合关注模型可解释性与翻译质量改进的研究者。
本文研究上下文感知机器翻译模型中注意力头在英译德、英译法任务中对代词消歧的影响。通过观察并调整可能影响代词预测的注意力分数，发现虽然部分注意力头关注了相关语义关系，但并非所有头都影响消歧能力。某些注意力头被模型低估，若增强其对特定关系的关注，有望提升性能。我们对最具潜力的注意力头进行微调，结果显示代词消歧准确率最高提升5个百分点，证明性能提升可固化到模型参数中。
原文摘要 · Abstract (English)
In this paper, we investigate the role of attention heads in Context-aware Machine Translation models for pronoun disambiguation in the English-to-German and English-to-French language directions. We analyze their influence by both observing and modifying the attention scores corresponding to the plausible relations that could impact a pronoun prediction. Our findings reveal that while some heads do attend the relations of interest, not all of them influence the models' ability to disambiguate pronouns. We show that certain heads are underutilized by the models, suggesting that model performance could be improved if only the heads would attend one of the relations more strongly. Furthermore, we fine-tune the most promising heads and observe the increase in pronoun disambiguation accuracy of up to 5 percentage points which demonstrates that the improvements in performance can be solidified into the models' parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。