通过归一化提升依存句法分析的参数效率
Dependency Parsing is More Parameter-Efficient with Normalization
- 对双线性打分引入归一化，缓解高方差输入导致的软最大值尖峰
- 在多语言数据上以更少参数和样本达到最优性能
- 适合关注模型压缩与高效训练的研究者
依存句法分析旨在推断自然语言结构，通常通过注意力机制建模词间交互，采用双线性打分。该机制类似于Transformer中的自注意力，为句子中每对词计算得分。然而，与Transformer注意力不同，双线性打分在取软最大值前未进行归一化。本文从理论和实证两方面表明，缺乏归一化会导致模型过度参数化：额外参数用于补偿双线性打分函数输入的高方差所引发的尖锐输出。我们主张通过评分归一化可显著提升双线性打分的效率。实验在多种语言的语义与句法依存分析任务以及非语言数据上的潜在图推理中展开，采用不同跳跃次数（k-hop）设置的解析器。使用N层堆叠的BiLSTM进行训练，并对比有无归一化时的性能表现。结果表明，归一化使模型能在更少样本和更少可训练参数下实现当前最优效果。
原文摘要 · Abstract (English)
Dependency parsing is the task of inferring natural language structure, often approached by modeling word interactions via attention through biaffine scoring. This mechanism works like self-attention in Transformers, where scores are calculated for every pair of words in a sentence. However, unlike Transformer attention, biaffine scoring does not use normalization prior to taking the softmax of the scores. In this paper, we provide theoretical evidence and empirical results revealing that a lack of normalization necessarily results in overparameterized parser models, where the extra parameters compensate for the sharp softmax outputs produced by high variance inputs to the biaffine scoring function. We argue that biaffine scoring can be made substantially more efficient by performing score normalization. We conduct experiments on semantic and syntactic dependency parsing in multiple languages, along with latent graph inference on non-linguistic data, using various settings of a $k$-hop parser. We train $N$-layer stacked BiLSTMs and evaluate the parser's performance with and without normalizing biaffine scores. Normalizing allows us to achieve state-of-the-art performance with fewer samples and trainable parameters. Code: https://github.com/paolo-gajo/EfficientSDP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。