用曲线表示句子，让语言模型更关注整体结构而非局部词预测。
Sentence Curve Language Models
- 用样条曲线作为句子的连续表示，控制点影响多个词
- 在IWSLT14和WMT14上超越现有扩散语言模型
- 训练稳定，无需知识蒸馏，适合追求结构建模的研究者
语言模型是现代AI系统的核心，扩散语言模型（DLMs）近期成为有力替代。两者均使用静态词嵌入表示输入与目标句子，但该方式对邻近词不敏感，侧重局部词预测而忽略全局句法结构。为此，我们提出一种连续句子表示——句子曲线，即通过控制点影响多个词的样条曲线。基于此，我们构建句子曲线语言模型（SCLM），将扩散模型扩展为预测句子曲线而非静态词嵌入。理论上，句子曲线预测具有正则化效应，促进全局结构建模，并可量化不同曲线类型的影响。实验表明，SCLM在IWSLT14和WMT14上达到当前最佳性能，训练稳定且无需复杂知识蒸馏，在LM1B上表现也优于离散扩散模型。
原文摘要 · Abstract (English)
Language models (LMs) are a central component of modern AI systems, and diffusion language models (DLMs) have recently emerged as a competitive alternative. Both paradigms rely on word embeddings not only to represent the input sentence, but also to represent the target sentence that backbone models are trained to predict. We argue that such static embedding of the target word is insensitive to neighboring words, encouraging locally accurate word prediction while global sentence structure is less emphasized. To address this, we propose a continuous sentence representation, termed sentence curve, defined as a spline curve whose control points affect multiple words in the sentence. Based on this representation, we introduce sentence curve language model (SCLM), which extends DLMs to predict sentence curves instead of the static word embeddings. We theoretically show that sentence curve prediction induces a regularization effect that promotes global structure modeling, and characterize how different sentence curve types affect this behavior. Empirically, SCLM achieves state-of-the-art performance among DLMs on IWSLT14 and WMT14, shows stable training without burdensome knowledge distillation, and demonstrates promising potential compared to discrete DLMs on LM1B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。