让注意力机制利用错位上下文作为有效特征，提升鲁棒性。
Indirect Attention: Turning Context Misalignment into a Feature
- 通过分析值特征噪声，发现错位上下文会引入超阈值噪声
- 提出间接注意力机制，显著提升错位场景下的性能表现
- 适合处理多模态或跨序列对齐困难的任务
注意力机制已成为现代深度学习架构的核心，通常键和值来自同一序列或表示。本文探讨一种非传统场景：键和值源自不同序列或模态。首先分析了在值特征含噪情况下的注意力行为，确定了一个关键噪声阈值，超过该阈值信号将显著退化。进一步将上下文（键、值）错位建模为值特征中的结构化噪声，发现此类错位引起的噪声可远超临界阈值，严重削弱标准注意力效果。为此，我们提出间接注意力（Indirect Attention），在上下文错位场景下通过间接推断相关性，实现更优性能。我们在一系列合成任务与真实应用中评估该方法，验证其在处理错位问题上的优越性。
原文摘要 · Abstract (English)
The attention mechanism has become a cornerstone of modern deep learning architectures, where keys and values are typically derived from the same underlying sequence or representation. This work explores a less conventional scenario, when keys and values originate from different sequences or modalities. Specifically, we first analyze the attention mechanism's behavior under noisy value features, establishing a critical noise threshold beyond which signal degradation becomes significant. Furthermore, we model context (key, value) misalignment as an effective form of structured noise within the value features, demonstrating that the noise induced by such misalignment can substantially exceed this critical threshold, thereby compromising standard attention's efficacy. Motivated by this, we introduce Indirect Attention, a modified attention mechanism that infers relevance indirectly in scenarios with misaligned context. We evaluate the performance of Indirect Attention across a range of synthetic tasks and real world applications, showcasing its superior ability to handle misalignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。