模型宽度线性增长时，深度可恒定，显著提升推理效率。
Depth-Width tradeoffs in Algorithmic Reasoning of Graph Tasks with Transformers
- 放宽宽度限制，发现常数深度即可解决多数图算法任务
- 线性宽度下，部分任务仅需常数深度，训练推理速度大幅提升
- 揭示了深度与宽度在图任务中的复杂权衡关系，适合追求效率的场景
Transformer 已成为机器学习的核心架构，在解决复杂算法问题（如图任务）方面表现突出。关键问题是：实现特定任务所需的最小 Transformer 规模是多少？已有研究显示，当嵌入维度（模型宽度）为次线性时，对数深度即足够。但本文关注的是：若宽度允许线性增长而深度保持固定，会发生什么？我们分析此情形，发现令人意外的结果：在宽度线性增长条件下，几乎所有图任务均可由常数深度模型解决。这表明适度增加宽度可大幅减少深度，从而在训练和推理时间上带来显著优势。对于其他任务，则需二次方宽度。我们的结果揭示了图算法中 Transformer 实现的复杂且引人深思的深度-宽度权衡景观。我们还通过实证研究验证了这种权衡，在多个任务中，更宽的模型达到与深层模型相当的精度，同时因硬件并行性而实现更快的训练与推理速度。
原文摘要 · Abstract (English)
Transformers have revolutionized the field of machine learning. In particular, they can be used to solve complex algorithmic problems, including graph-based tasks. In such algorithmic tasks a key question is what is the minimal size of a transformer that can implement the task. Recent work has begun to explore this problem for graph-based tasks, showing that for sub-linear embedding dimension (i.e., model width) logarithmic depth suffices. However, an open question, which we address here, is what happens if width is allowed to grow linearly, while depth is kept fixed. Here we analyze this setting, and provide the surprising result that with linear width, constant depth suffices for solving a host of graph-based problems. This suggests that a moderate increase in width can allow much shallower models, which are advantageous in terms of inference and train time. For other problems, we show that quadratic width is required. Our results demonstrate the complex and intriguing landscape of transformer implementations of graph-based algorithms. We empirically investigate these trade-offs between the relative powers of depth and width and find tasks where wider models have the same accuracy as deep models, while having much faster train and inference time due to parallelizable hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。