让Transformer循环推理可变深度,适应不同算力预算。
LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
- 通过捷径一致性训练,让不同循环长度的输出保持一致。
- 在算力受限时仍保持强性能,算力增加则自动提升效果。
- 适合需要灵活算力控制的大型语言模型应用。
循环Transformer已成为语言领域高效且强大的推理模型。近期研究表明,这类模型在算法和推理任务上表现优异,表明循环结构具有潜在的隐式推理偏好。然而,先前方法在训练和推理中固定循环次数,未解决模型能否在不同计算预算下灵活调整计算深度的问题。本文提出LoopFormer,一种在可变长度轨迹上训练的循环Transformer,实现基于预算的推理。核心贡献是捷径一致性训练策略,使不同长度轨迹对齐,确保短循环产生有意义表征,长循环持续优化。每个循环条件于当前时间与步长,使表征在不同长度轨迹间稳定演化而非漂移或停滞。实验表明,即便在严苛算力约束下,LoopFormer在语言建模与推理基准上仍表现稳健,且能随算力增加平稳扩展。结果表明,循环Transformer天生适合自适应语言建模,为可控、预算感知的大模型开辟新路径。
原文摘要 · Abstract (English)
Looped Transformers have emerged as an efficient and powerful class of models for reasoning in the language domain. Recent studies show that these models achieve strong performance on algorithmic and reasoning tasks, suggesting that looped architectures possess an inductive bias toward latent reasoning. However, prior approaches fix the number of loop iterations during training and inference, leaving open the question of whether these models can flexibly adapt their computational depth under variable compute budgets. We introduce LoopFormer, a looped Transformer trained on variable-length trajectories to enable budget-conditioned reasoning. Our core contribution is a shortcut-consistency training scheme that aligns trajectories of different lengths, ensuring that shorter loops yield informative representations while longer loops continue to refine them. LoopFormer conditions each loop on the current time and step size, enabling representations to evolve consistently across trajectories of varying length rather than drifting or stagnating. Empirically, LoopFormer demonstrates robust performance on language modeling and reasoning benchmarks even under aggressive compute constraints, while scaling gracefully with additional budget. These results show that looped Transformers are inherently suited for adaptive language modeling, opening a path toward controllable and budget-aware large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。