分离输入输出表征，让语言模型更懂罕见词
Leviathan: Decoupling Input and Output Representations in Language Models
- 用连续映射替代输入嵌入矩阵，解耦表征与词汇判别
- 在1.2B模型上降低9%困惑度，稀有词困惑度下降81%
- 适合关注长尾词汇表现或追求高效训练的研究者
现代语言模型使用单一矩阵实现输入嵌入和输出投影，将词表征与词汇判别目标耦合。本文提出Leviathan，一种采用可学习嵌入向量化（LEV）的Transformer架构，以连续映射方式将词索引转换为嵌入向量，取代传统输入嵌入矩阵。输出头保持未绑定，参数仅增加0.2%。在相同Transformer主干下，于The Pile数据集上对200M至1.2B参数规模进行对比实验，结果表明Leviathan在训练过程中持续优于标准绑接嵌入基线。在1.2B规模下，验证困惑度降低9%，达到基线最终损失所需训练样本减少2.1倍，且在全部六个下游任务中表现更优，包括LAMBADA困惑度下降30%。频率分层分析显示，性能提升主要集中于稀有词，其困惑度下降81%，而高频词几乎无提升。
原文摘要 · Abstract (English)
Modern language models use a single matrix for input embedding and output projection. This couples two distinct objectives: token representation and discrimination over a vocabulary. This work introduces Leviathan, a Transformer architecture that replaces the input embedding matrix with learned embedding vectorization (LEV), a compact continuous mapping from token indices to embeddings. Leviathan's output head remains untied for a parameter increase of as low as 0.2%. Under controlled comparisons with identical Transformer backbones, Leviathan consistently improves language modeling performance over standard tied-embedding baselines across a 200M-1.2B parameter regime on The Pile with gains that grow during training. At 1.2B scale, Leviathan reduces validation perplexity by 9%, requires $2.1\times$ fewer training tokens to reach the tied baseline's final loss, and improves on all six downstream benchmarks evaluated, including a 30% reduction in LAMBADA perplexity. Frequency-stratified analysis reveals gains to be concentrated in rare tokens, where continuous parameterization reduces perplexity by 81%, falling to near zero for the most frequent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。