模型架构与推理计算可分离，提升效率且不损失预测能力
Dissociating model architectures from inference computations
- 将模型架构与推理计算解耦，通过迭代推理结构化上下文访问
- 基于Transformer的自回归模型在迭代中实现层级时间因子分解
- 推理计算量减少，仍保持原有预测性能，适合高效部署场景
Parr等（2025）研究了自回归模型与深层时间模型在非马尔可夫序列建模中的差异。本文强调需将模型架构（即预测分布的因式分解方式）与推理时的计算过程相分离。我们证明，通过在迭代推理中结构化上下文访问，自回归模型可模拟深层时间计算。以一个基于下一个词预测训练的Transformer为例，发现在迭代推理中引入层级时间因子分解，可在减少计算量的同时维持预测能力。这表明预测构建与优化过程并不必然绑定于底层模型架构。
原文摘要 · Abstract (English)
Parr et al., 2025 examines how auto-regressive and deep temporal models differ in their treatment of non-Markovian sequence modelling. Building on this, we highlight the need for dissociating model architectures, i.e., how the predictive distribution factorises, from the computations invoked at inference. We demonstrate that deep temporal computations are mimicked by autoregressive models by structuring context access during iterative inference. Using a transformer trained on next-token prediction, we show that inducing hierarchical temporal factorisation during iterative inference maintains predictive capacity while instantiating fewer computations. This emphasises that processes for constructing and refining predictions are not necessarily bound to their underlying model architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。