Transformer可模拟常数深度算术电路，用平均注意力实现计算
Average Attention Transformers and Arithmetic Circuits
- 用平均注意力替代前馈网络，构建基于算术电路的Transformer
- 能计算常数深度、无界加法与乘法的算术电路函数
- 适用于实数、有理数等环域，理论意义强
我们分析了Transformer编码器作为向量序列到序列函数的计算能力。研究表明，若将算术电路作为输入，平均硬注意力可模拟此类电路。所模拟的电路家族具有常数深度，且允许无界加法、二元乘法和符号门。本文使用的Transformer以算术电路替代传统前馈网络；在典型平均注意力下，其计算函数也属于同一类电路家族。该结论适用于实数、有理数及二者之间的任意环。
原文摘要 · Abstract (English)
We analyse the computational power of transformer encoders as sequence-to-sequence functions on vectors. We show that average hard attention can be used to simulate arithmetic circuits if they are given as an input to an encoder. The circuit families that can be simulated this way have constant depth while using unbounded addition, binary multiplication and sign gates. The transformers we use have arithmetic circuits instead of feed-forward networks. With typical average attention the functions they compute are also computed by the same class of circuit families. Our results hold for transformers over the reals, rationals and any ring in between the two.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。