Transformer能否学会n-gram语言模型？实验证明它在特定情况下表现更优。
Can Transformers Learn $n$-gram Language Models?
- 用随机n-gram模型测试Transformer学习能力，对比传统估计方法
- 共享参数的n-gram模型中，Transformer优于add-λ平滑等经典方法
- 揭示理论可表示性与实际学习能力间的差距，适合对模型机制感兴趣者
大量理论研究描述了Transformer表示形式语言的能力。然而，由于架构、学习算法和训练数据之间的复杂交互，将理论结果与实际性能联系起来并不直接。为检验理论下界是否意味着形式语言的可学习性，我们转向近期将Transformer与n-gram语言模型关联的研究。我们考察了Transformer学习两类随机n-gram语言模型的能力：一类是具有任意下一符号概率的模型，另一类是这些概率由共享参数定义的模型。结果显示，在前者中，add-λ平滑等经典估计技术优于Transformer；而在后者中，Transformer表现更好，甚至超越专为学习n-gram模型设计的方法。
原文摘要 · Abstract (English)
Much theoretical work has described the ability of transformers to represent formal languages. However, linking theoretical results to empirical performance is not straightforward due to the complex interplay between the architecture, the learning algorithm, and training data. To test whether theoretical lower bounds imply \emph{learnability} of formal languages, we turn to recent work relating transformers to $n$-gram language models (LMs). We study transformers' ability to learn random $n$-gram LMs of two kinds: ones with arbitrary next-symbol probabilities and ones where those are defined with shared parameters. We find that classic estimation techniques for $n$-gram LMs such as add-$λ$ smoothing outperform transformers on the former, while transformers perform better on the latter, outperforming methods specifically designed to learn $n$-gram LMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。