通过自适应循环结构，让Transformer更好处理未知长度的算法任务。
Looped Transformers for Length Generalization
- 用可变步数的循环Transformer替代固定结构
- 在多种迭代任务上实现跨长度泛化，性能显著提升
- 适合需要处理可变输入长度的算法推理场景
近期研究表明，从零训练的Transformer能成功解决加法、奇偶性判断等算术与算法任务。尽管它们对相同长度的未见输入具有良好泛化能力，但在处理未知长度输入时表现不佳。本文提出使用具有自适应步数的循环Transformer，显著提升长度泛化能力。研究聚焦于具有已知迭代解法的任务，涉及多次RASP-L操作——一种可用有限尺寸Transformer表达的长度泛化操作。通过所提出的训练算法，循环Transformer学习到了多种任务的高长度泛化解法。
原文摘要 · Abstract (English)
Recent work has shown that Transformers trained from scratch can successfully solve various arithmetic and algorithmic tasks, such as adding numbers and computing parity. While these Transformers generalize well on unseen inputs of the same length, they struggle with length generalization, i.e., handling inputs of unseen lengths. In this work, we demonstrate that looped Transformers with an adaptive number of steps significantly improve length generalization. We focus on tasks with a known iterative solution, involving multiple iterations of a RASP-L operation - a length-generalizable operation that can be expressed by a finite-sized Transformer. We train looped Transformers using our proposed learning algorithm and observe that they learn highly length-generalizable solutions for various tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。