揭示循环Transformer函数逼近能力瓶颈并提出时间步编码增强方法。
On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding
- 通过序列到序列函数的连续性模定义，分析循环Transformer的逼近速率。
- 实验证明增加循环次数可提升性能，时间步编码带来额外增益。
- 适用于需高效推理的序列建模任务，如长序列生成与逻辑推理。
循环Transformer在参数效率、计算能力和泛化性方面对推理任务具有优势，但其在函数逼近方面的表达能力尚未得到充分研究。本文通过定义序列到序列函数的连续性模，建立了循环Transformer的逼近速率理论。分析揭示了循环结构特有的局限性：需引入依赖时间步编码的缩放参数以克服该限制。实验验证了理论结果，表明增加循环次数可提升性能，且时间步编码进一步增强了模型表现。
原文摘要 · Abstract (English)
Looped Transformers provide advantages in parameter efficiency, computational capabilities, and generalization for reasoning tasks. However, their expressive power regarding function approximation remains underexplored. In this paper, we establish the approximation rate of Looped Transformers by defining the modulus of continuity for sequence-to-sequence functions. This reveals a limitation specific to the looped architecture. That is, the analysis prompts the incorporation of scaling parameters for each loop, conditioned on timestep encoding. Experiments validate the theoretical results, showing that increasing the number of loops enhances performance, with further gains achieved through the timestep encoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。