用偏微分方程让注意力随时间演化，解决长序列建模难题
Continuous-Time Attention: PDE-Guided Mechanisms for Long-Sequence Transformers
- 将偏微分方程引入注意力机制，让权重在伪时间上动态演化
- 在长序列任务中显著提升性能，优化过程更稳定，远距离依赖衰减变慢
- 适合处理超长序列的NLP与时序建模，对理论和工程均有启发
我们提出一种新框架——连续时间注意力（Continuous-Time Attention），将偏微分方程（PDE）融入Transformer的注意力机制，以应对极长输入序列的挑战。不同于传统的静态注意力矩阵，该方法通过扩散、波动或反应-扩散动力学，使注意力权重在伪时间维度上动态演化。该机制系统性地平滑局部噪声，增强长距离依赖，并稳定梯度传播。理论上，分析表明基于PDE的注意力能带来更优的优化景观，且远距离交互的衰减速率由指数型变为多项式。实验在多种任务上验证了该方法相较于标准及专用长序列Transformer模型的一致优势。结果表明，基于PDE的公式可为注意力机制注入连续时间动态与全局一致性。
原文摘要 · Abstract (English)
We propose a novel framework, Continuous_Time Attention, which infuses partial differential equations (PDEs) into the Transformer's attention mechanism to address the challenges of extremely long input sequences. Instead of relying solely on a static attention matrix, we allow attention weights to evolve over a pseudo_time dimension via diffusion, wave, or reaction_diffusion dynamics. This mechanism systematically smooths local noise, enhances long_range dependencies, and stabilizes gradient flow. Theoretically, our analysis shows that PDE_based attention leads to better optimization landscapes and polynomial rather than exponential decay of distant interactions. Empirically, we benchmark our method on diverse experiments_demonstrating consistent gains over both standard and specialized long sequence Transformer variants. Our findings highlight the potential of PDE_based formulations to enrich attention mechanisms with continuous_time dynamics and global coherence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。