提出新型注意力机制，让多任务模型更高效且更准确。
Efficient Inter-Task Attention for Multitask Transformer Models
- 设计可变形跨任务自注意力，减少计算冗余。
- 在两个数据集上实现推理延迟和计算量降低一个数量级。
- 提升单任务精度最高达7.4%，适合多任务视觉任务研究者。
在计算机视觉及更广泛的深度学习领域，Transformer架构已被证实是众多应用的最优选择。然而，在多任务学习中，由于任务数量增加导致查询数量显著上升，传统的多头注意力机制面临计算瓶颈，尤其受限于实际硬件条件。这是因为注意力矩阵的规模随任务数量呈平方级增长（假设各任务查询数相近）。为此，我们提出一种新型的可变形跨任务自注意力机制，有效提升了多任务特征图间信息聚合的效率。在NYUD-v2与PASCAL-Context数据集上的实验表明，该方法实现了计算量（FLOPs）与推理延迟均降低一个数量级，同时各项任务的预测精度最高提升7.4%。
原文摘要 · Abstract (English)
In both Computer Vision and the wider Deep Learning field, the Transformer architecture is well-established as state-of-the-art for many applications. For Multitask Learning, however, where there may be many more queries necessary compared to single-task models, its Multi-Head-Attention often approaches the limits of what is computationally feasible considering practical hardware limitations. This is due to the fact that the size of the attention matrix scales quadratically with the number of tasks (assuming roughly equal numbers of queries for all tasks). As a solution, we propose our novel Deformable Inter-Task Self-Attention for Multitask models that enables the much more efficient aggregation of information across the feature maps from different tasks. In our experiments on the NYUD-v2 and PASCAL-Context datasets, we demonstrate an order-of-magnitude reduction in both FLOPs count and inference latency. At the same time, we also achieve substantial improvements by up to 7.4% in the individual tasks' prediction quality metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。