用动态值注意力替换传统Transformer，单头就能高效完成任务。
Transformer Reconstructed with Dynamic Value Attention
- 为每个查询动态生成对应值，不再使用静态共享值
- 训练时间减少37.6%，且学习能力更强
- 适合追求效率与轻量化的模型部署场景
自2017年Transformer提出以来，虽有诸多优化工作，但其核心结构仍保持不变，忽视了其一个关键内在缺陷：每个注意力头中所有查询共享相同的静态值。尽管多头注意力试图缓解此问题，但头数受限于计算复杂度。本文提出一种方法，为每个查询动态生成对应的值，从而可完全去除冗余的多头结构，仅保留单一注意力头。相应地，后续的前馈网络也可彻底省去，因为每个重构后的嵌入已获取足够多的有效信息，远超上下文范围。实验表明，采用单头动态值注意力（DVA）的模型相比原Transformer可节省37.6%训练时间，同时提升学习能力。
原文摘要 · Abstract (English)
Since transformer was firstly published in 2017, several works have been proposed to optimize it. However, the major structure of transformer remains unchanged, ignoring one of its main intrinsic limitations, which is the same static value is used for every query in a head. Transformer itself tries to solve this problem by implementing multi-head attentions, yet the number of heads is limited by complexity. I propose a method to decide a value for each query dynamically, which could cut down all the redundant heads, keeping only one. Consequently, the following feed forward network could be cut down entirely, as each revised embedding has already fetched enough useful values far beyond the context. As a result, a single-head Dynamic Value Attention (DVA) is all you need in a transformer. According to the experiment, DVA may save 37.6% training time than the original transformer meanwhile increasing the learning capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。