改进注意力机制，让模型更准更稳地处理长文本。
DINT Transformer
- 用差分-积分机制融合全局重要性信息，增强上下文理解。
- 在长文本建模和关键信息检索任务中准确率显著提升。
- 设计更稳定，适合需要高鲁棒性的实际应用。
DIFF Transformer 通过引入差分注意力机制提升了局部注意力的鲁棒性，但存在两大缺陷：缺乏全局上下文建模能力，难以识别全局重要标记；且注意力矩阵缺少严格的行归一化，导致数值不稳定。为此，本文提出 DINT Transformer，通过引入差分-积分机制，在计算全局重要性得分并融入注意力矩阵的同时，采用统一参数设计强制实现行归一化，从而增强对全局依赖的捕捉能力并提升数值稳定性。实验表明，DINT Transformer 在长上下文语言建模与关键信息检索等实际应用中均表现出更高精度与更强鲁棒性，展现出高效且有前景的架构潜力。
原文摘要 · Abstract (English)
DIFF Transformer addresses the issue of irrelevant context interference by introducing a differential attention mechanism that enhances the robustness of local attention. However, it has two critical limitations: the lack of global context modeling, which is essential for identifying globally significant tokens, and numerical instability due to the absence of strict row normalization in the attention matrix. To overcome these challenges, we propose DINT Transformer, which extends DIFF Transformer by incorporating a differential-integral mechanism. By computing global importance scores and integrating them into the attention matrix, DINT Transformer improves its ability to capture global dependencies. Moreover, the unified parameter design enforces row-normalized attention matrices, improving numerical stability. Experimental results demonstrate that DINT Transformer excels in accuracy and robustness across various practical applications, such as long-context language modeling and key information retrieval. These results position DINT Transformer as a highly effective and promising architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。