用自学习语法结构提升模型理解与生成能力
Leveraging Grammar Induction for Language Understanding and Generation
- 无监督学习语法结构，直接融入Transformer的注意力机制
- 在机器翻译和理解任务中超越原始Transformer和其他外部解析器增强模型
- 适用于从零训练到预训练场景，尤其适合追求语法感知的模型
近年来，语法归纳取得了显著进展。然而，如何利用归纳出的语法结构提升下游任务的实际性能仍不明确。本文提出一种无监督的语法归纳方法，用于语言理解和生成。我们构建了一个语法解析器，用于推导成分结构和依存关系，并在无需额外句法标注的情况下，与下游任务联合训练。所获得的语法特征随后被引入Transformer作为语法掩码，引导自注意力机制。我们在多个机器翻译和自然语言理解任务上评估并应用该方法。实验结果表明，该方法在性能上优于原始Transformer及其他依赖外部解析器的模型。研究进一步验证了显式建模文本语法结构对神经网络的有效性，且在从零训练和预训练两种场景下均表现优异。
原文摘要 · Abstract (English)
Grammar induction has made significant progress in recent years. However, it is not clear how the application of induced grammar could enhance practical performance in downstream tasks. In this work, we introduce an unsupervised grammar induction method for language understanding and generation. We construct a grammar parser to induce constituency structures and dependency relations, which is simultaneously trained on downstream tasks without additional syntax annotations. The induced grammar features are subsequently incorporated into Transformer as a syntactic mask to guide self-attention. We evaluate and apply our method to multiple machine translation tasks and natural language understanding tasks. Our method demonstrates superior performance compared to the original Transformer and other models enhanced with external parsers. Experimental results indicate that our method is effective in both from-scratch and pre-trained scenarios. Additionally, our research highlights the contribution of explicitly modeling the grammatical structure of texts to neural network models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。