用树形结构替代线性Transformer，提升效率与可扩展性
TreeCoders: Trees of Transformers
- 将Transformer块构建成k叉树，通过分类器选择路径
- 在多种数据集上76%情况下优于同等规模的线性模型
- 支持稀疏激活和分布式计算，适合大规模应用
本文提出TreeCoders，一种新型Transformer树架构。我们摒弃传统线性Transformer，改用完整的k-ary树结构。Transformer块作为节点，通用分类器学习选择最优子节点，并将标记序列路由至特定叶节点。选择器置于Transformer块外部，使架构兼容性更强且无需额外修改。此外，树搜索的对数复杂度支持稀疏节点激活。我们在一系列仅解码器的树形Transformer上验证了该方法，在多种语言数据集上表现良好。研究显示，所提出的树形Transformer模型在广泛树结构下，76%的情况下优于同等规模的线性Transformer。同时，该模型天然支持分布式实现。
原文摘要 · Abstract (English)
In this paper, we introduce TreeCoders, a novel family of transformer trees. We moved away from traditional linear transformers to complete k-ary trees. Transformer blocks serve as nodes, and generic classifiers learn to select the best child and route the sequence of tokens to a specific leaf. The selectors, moved outside the transformer blocks, allow for the use of a variety of architecture without further modifications. Furthermore, our proposed architecture supports sparse node activation due to the logarithmic complexity of a tree search. We validate our idea by testing a series of decoder-only tree transformers, achieving competitive results across a diverse range of language datasets. Our study demonstrates that the proposed tree transformer model outperforms a size-equivalent linear transformer model 76\% of the time over a wide range of tree architectures. Furthermore, our proposed model naturally lends itself to distributed implementation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。