用注意力机制实现多任务计算，揭示头数与精度的权衡关系
Attention-based representations for multi-task computation
- 通过多头注意力构造支持多任务的向量表示
- 单头需指数级更高维度或精度，多头可突破此瓶颈
- 适用于需要高效多任务建模的场景，如逻辑运算建模
多头注意力层可生成支持多个下游任务的向量表示。本文在两个具体多任务场景中建立了所需头数的理论边界。第一种场景要求向量表示能支持线性预测器同时计算列表中的最小值和最大值，已知两个小维度、低精度的注意力头即可实现；我们证明单个头需指数级更高的嵌入维度或精度。第二种场景要求多项式阈值函数能计算长度为 $n$ 位的异或（XOR）结果，该问题与 $n=2$ 时的线性情形类似，因异或可通过编码与、或运算的向量表示由线性函数实现。我们发现 $n$ 位异或要求头数与多项式阶次的乘积至少为 $n$，并构造出达到此下界的多头注意力结构。该结论可推广至任意对称布尔函数，边界由阈值度决定。
原文摘要 · Abstract (English)
Multi-head attention layers produce vector representations that support multiple downstream tasks. We establish bounds on the number of heads required in two simple and concrete multi-task scenarios. In the first scenario, a vector representation is sought so that linear predictors can compute both the smallest and largest numbers in a given list. In this case, it is known two attention heads with small embedding dimension and bit precision level suffice. We prove that a single attention head requires exponentially higher embedding dimension or precision level. In the second scenario, a vector representation is sought so that a polynomial threshold function can compute the XOR of a given string of $n$ bits. This scenario is analogous to the first one for $n=2$, since XOR is readily computed by a linear function using a vector representation that encodes both the AND and the OR of the two bits. We observe that $n$-bit XOR requires the product of the number of heads and the polynomial degree to be at least $n$, and we construct multi-head attention layers that match this lower bound. These results generalize to arbitrary (symmetric) Boolean functions, where the bound is given in terms of the threshold degree.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。