通过语法平滑缓解语言模型的低频词偏差与表征聚集问题
Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing
- 引入语法先验，调整最大似然目标函数，使相似语法结构的词共享学习信号
- 在低频英语词上表现提升，模型表征各向异性显著降低
- 适合关注模型泛化性与表征质量的研究者
语言模型在预训练中依赖词频信息，最大化词元的似然概率，导致对低频词泛化能力差。此外，最大似然训练引发表征各向异性：词元表示集中在高维锥形区域，未充分利用表征空间。本文提出一种量化语言模型频率偏差的方法，基于词级频率评估句级困惑度。进一步提出语法平滑（Syntactic Smoothing）方法，在预训练中对词元表示施加语法先验，调整最大似然目标函数，将学习信号分布至语法相似的词元。该方法在低频词上取得更好性能，并降低模型各向异性。实验表明，模型各向异性程度与其频率偏差呈正相关。
原文摘要 · Abstract (English)
Language models strongly rely on frequency information because they maximize the likelihood of tokens during pre-training. As a consequence, language models tend to not generalize well to tokens that are seldom seen during training. Moreover, maximum likelihood training has been discovered to give rise to anisotropy: representations of tokens in a model tend to cluster tightly in a high-dimensional cone, rather than spreading out over their representational capacity. Our work introduces a method for quantifying the frequency bias of a language model by assessing sentence-level perplexity with respect to token-level frequency. We then present a method for reducing the frequency bias of a language model by inducing a syntactic prior over token representations during pre-training. Our Syntactic Smoothing method adjusts the maximum likelihood objective function to distribute the learning signal to syntactically similar tokens. This approach results in better performance on infrequent English tokens and a decrease in anisotropy. We empirically show that the degree of anisotropy in a model correlates with its frequency bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。