发现自回归语言模型对文本方向不敏感，可能遗漏语言的时序本质。
Reversal Invariance in Autoregressive Language Models
- 提出自回归语言建模具有反转不变性，正向与反向文本训练效果相同。
- 实验证明反向训练模型性能与正向相当，说明当前预训练目标缺乏方向感知。
- 主张未来模型应显式建模语言的时间箭头，提升对语义因果的理解能力。
我们形式化了自回归语言建模（CLM）目标的一种结构性特性：反转不变性。该性质表明，下一个词预测损失对语料库及其反转版本赋予相同的似然值，意味着标准CLM预训练是方向盲的。这一对称性解释了为何在反向文本上训练的模型仍能达到与正向文本训练模型相当的性能，尽管人类语言和推理具有固有的时间不对称性。我们认为这种不变性并非无害的副产品，而是当前预训练目标的局限。若自然语言包含时序依赖关系（如语音、形态或因果），对称目标可能无法捕捉这些特征。因此，我们主张从时间不对称性的视角重新审视预训练，推动未来研究开发能显式建模语言方向性、同时保持标准语言建模能力的损失函数与架构。
原文摘要 · Abstract (English)
We formalize a structural property of the causal (autoregressive) language modeling (CLM) objective: reversal invariance. Formally, the next-token prediction loss assigns identical likelihood to a corpus and its reversal, implying that standard CLM pretraining is direction-blind. This symmetry explains why models trained on reversed text can achieve comparable performance to those trained on forward text, despite the inherently time-asymmetric nature of human language and reasoning. We argue that this invariance represents a limitation of current pretraining objectives rather than a benign artifact. If natural language encodes directional dependencies - phonological, morphological, or causal - a symmetric objective may fail to capture them. We therefore propose viewing pretraining through the lens of temporal asymmetry, motivating future work on loss functions and architectures that explicitly model the arrow of language while retaining standard language modeling capacity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。