用依存图增强Transformer，提升语法泛化能力。
GiLT: Augmenting Transformer Language Models with Dependency Graphs
- 通过动态构建的依存图调节注意力权重
- 在语法泛化上优于基线模型，困惑度相当
- 可从预训练模型微调，提升下游任务表现
将语言学结构融入Transformer能有效提升语言模型的语法泛化性能。以往研究多聚焦于句法树结构，特别是成分树。本文提出图注入层语言模型（GiLT），利用依存图增强Transformer语言模型。与多数工作不同，GiLT不添加额外结构标记，而是通过增量构建的依存图提取特征，动态调节Transformer中的注意力权重。实验表明，使用语义依存图的GiLT在语法泛化上优于基线模型，同时保持相近的困惑度。此外，GiLT可从预训练模型微调，在下游任务中取得更优表现。代码已开源：https://github.com/cookie-pie-oops/GiLT-LM。
原文摘要 · Abstract (English)
Augmenting Transformers with linguistic structures effectively enhances the syntactic generalization performance of language models. Previous work in this direction focuses on syntactic tree structures of languages, in particular constituency tree structures. We propose Graph-Infused Layers Transformer Language Model (GiLT) which leverages dependency graphs for augmenting Transformer language models. Unlike most previous work, GiLT does not insert extra structural tokens in language modeling; instead, it injects structural information into language modeling by modulating attention weights in the Transformer with features extracted from the dependency graph that is incrementally constructed along with token prediction. In our experiments, GiLT with semantic dependency graphs achieves better syntactic generalization while maintaining competitive perplexity in comparison with Transformer language model baselines. In addition, GiLT can be finetuned from a pretrained language model to achieve improved downstream task performance. Our code is released at https://github.com/cookie-pie-oops/GiLT-LM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。