多任务学习中共享注意力会失效，新方法让注意力随结果变化
Why shared attention vectors fail: a case for outcome-indexed tuning
- 用结果索引的注意力矩阵替代全局共享向量
- 实验显示新方法能稳定收敛，共享向量则会塌陷到边界
- 适合研究多目标学习与注意力机制的学者参考
在学习中的维度注意力通常以全局共享的注意力向量实现，每个刺激维度对应一个标量，模型通过误差梯度下降学习这些标量，使预测特征更显著。我们发现，在多结果学习（即模型需预测多个输出）条件下，这种共享向量会变得不稳定，最终塌陷至取值边界，导致模型无法学习有意义的注意力调优。为此，我们提出一种结果索引的注意力矩阵，将全局共享的注意力调优转换为按结果索引的表示形式。我们分析了共享向量不稳定的成因，并推导出其成立条件。三个合成实验验证了所提注意力矩阵的有效性：它们能收敛到有意义的表示，而共享注意力向量则无法做到。结果表明，结果索引的注意力矩阵是梯度驱动注意力过程的一般性解决方案，可提升多结果学习场景下的建模能力。
原文摘要 · Abstract (English)
Dimensional attention in learning is often implemented as a globally shared attention vector, where each stimulus dimension corresponds to a single scalar. These scalars are learned by models through gradient-descent on error, where predictive features acquire more salience. We show that under multi-outcome learning, where models predict more than one outcome, this shared vector becomes unstable; it collapses to its bounds and prevents the models from learning meaningful attentional tunings for learning and generalization. We address this by introducing an outcome-indexed attentional matrix that converts globally shared attentional tuning into an outcome-indexed representation. We present an analysis of the unstable shared vectors and derive the conditions under which it holds. Empirically, three synthetic experiments benchmark the proposed attention matrices and show that they converge to meaningful representations, something shared attention vectors fail to do. These results suggest that outcome-indexed attentional matrices are a general fix for gradient-based attentional processes, which improves models of learning under multi-outcome conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。