大语言模型隐式将语言建模为连续函数，突破了传统离散序列的局限。
Language Models Are Implicitly Continuous
- 将Transformer扩展为连续输入输出空间的函数逼近器
- Llama2至Mistral等主流模型均表现出连续性特征
- 为理解模型推理机制提供新视角，适合研究者与架构设计者
语言通常被建模为离散序列，但最成功的语言建模方法——神经网络——本质上是连续平滑的函数逼近器。本文表明，基于Transformer的大语言模型隐式地将句子表示为定义在连续输入空间上的连续时间函数。这一现象出现在多数顶尖大模型中，包括Llama2、Llama3、Phi3、Gemma、Gemma2和Mistral，暗示这些模型对语言的理解方式与人类存在根本差异。本工作形式化扩展了Transformer，以捕捉输入与输出空间中的时空连续性细微差别。结果挑战了传统对大模型理解语言方式的认知，具有多重语言学与工程意义。
原文摘要 · Abstract (English)
Language is typically modelled with discrete sequences. However, the most successful approaches to language modelling, namely neural networks, are continuous and smooth function approximators. In this work, we show that Transformer-based language models implicitly learn to represent sentences as continuous-time functions defined over a continuous input space. This phenomenon occurs in most state-of-the-art Large Language Models (LLMs), including Llama2, Llama3, Phi3, Gemma, Gemma2, and Mistral, and suggests that LLMs reason about language in ways that fundamentally differ from humans. Our work formally extends Transformers to capture the nuances of time and space continuity in both input and output space. Our results challenge the traditional interpretation of how LLMs understand language, with several linguistic and engineering implications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。