长序列模型无法同时做到高效、紧凑和强记忆。
The Impossibility Triangle of Long-Context Modeling
- 提出长序列模型的三难困境：效率、紧凑性、记忆能力不可兼得。
- 理论证明：高效紧凑模型最多记住O(poly(d)/log V)个关键信息。
- 实验证实52种架构均困在三角内，无一突破极限。
我们揭示并证明了长序列模型的根本性权衡：任何模型都无法同时实现（1）每步计算与序列长度无关（效率），（2）状态大小与序列长度无关（紧凑性），以及（3）回忆与序列长度成比例的历史事实（记忆能力）。我们在一个统一的在线序列处理器框架下形式化这一权衡，涵盖Transformer、状态空间模型、线性递归网络及其混合体。利用数据处理不等式和法诺不等式，我们证明：满足效率与紧凑性的模型，最多能从任意长序列中回忆出O(poly(d)/log V)个键值对，其中d为模型维度，V为词汇表大小。我们将2026年3月前发表的52种架构归入该三角，发现每种均最多满足两个属性，混合架构则在三角内部形成连续轨迹。在合成关联回忆任务上的实验验证了理论界限：实际记忆容量严格低于信息论极限，且无模型能逃脱该三角。
原文摘要 · Abstract (English)
We identify and prove a fundamental trade-off governing long-sequence models: no model can simultaneously achieve (i) per-step computation independent of sequence length (Efficiency), (ii) state size independent of sequence length (Compactness), and (iii) the ability to recall a number of historical facts proportional to sequence length (Recall). We formalize this trade-off within an Online Sequence Processor abstraction that unifies Transformers, state space models, linear recurrent networks, and their hybrids. Using the Data Processing Inequality and Fano's Inequality, we prove that any model satisfying Efficiency and Compactness can recall at most O(poly(d)/log V) key-value pairs from a sequence of arbitrary length, where d is the model dimension and V is the vocabulary size. We classify 52 architectures published before March 2026 into the triangle, showing that each achieves at most two of the three properties and that hybrid architectures trace continuous trajectories in the interior. Experiments on synthetic associative recall tasks with five representative architectures validate the theoretical bound: empirical recall capacity lies strictly below the information-theoretic limit, and no architecture escapes the triangle.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。