通过统一框架评估句法模型，优化语言生成与推理效率。
A Systematic Study of Compositional Syntactic Transformer Language Models
- 基于成分句法树构建自底向上的组合模型
- 在多项任务中验证了句法信息提升泛化能力
- 适合关注句法结构与模型效率的研究者
句法语言模型（SLMs）通过建模线性化的句法解析树来增强Transformer，引入句法先验。本文聚焦基于成分句法树的组合式SLMs，其显式实现成分表示的自底向上组合。我们识别了现有组合式SLMs的设计关键点，提出一个涵盖现有模型与新变体的统一框架。在语言建模、句法泛化、摘要生成、对话理解和推理效率等任务上，对框架内所有变体进行了全面实证评估。基于实验结果，提出了多项组合式SLMs设计建议。代码已开源。
原文摘要 · Abstract (English)
Syntactic language models (SLMs) enhance Transformers by incorporating syntactic biases through the modeling of linearized syntactic parse trees alongside surface sentences. This paper focuses on compositional SLMs that are based on constituency parse trees and contain explicit bottom-up composition of constituent representations. We identify key aspects of design choices in existing compositional SLMs and propose a unified framework encompassing both existing models and novel variants. We conduct a comprehensive empirical evaluation of all the variants in our framework across language modeling, syntactic generalization, summarization, dialogue, and inference efficiency. Based on the experimental results, we make multiple recommendations on the design of compositional SLMs. Our code is released at https://github.com/zhaoyd1/compositional_SLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。