改进注意力机制,让模型更清楚地记住证据重复次数。
What Softmax Throws Away: Mass-Aware Attention for Evidence Accumulation

- 提出质量感知注意力,通过Lp归一化改变重复证据的聚合方式。
- 在12个实验中11个提升未来链接预测性能,平均恢复能力提高4.49%。
- 无需额外参数或标签,适用于动态图、知识图谱等多种场景。
高任务性能未必反映模型是否保留了预测相关结构信息。例如,时序图模型虽能实现高未来链接AUC,但基本图统计仍难以从同一表示中恢复。我们发现标准注意力中的加权平均是问题根源:当证据模式重复时,分子分母同比增长,导致不同累积证据量产生相同聚合结果。为此提出质量感知注意力(MAA),将标准L1归一化推广至Lp族。在重复情形下,分子与分母以不同速率增长,使表示幅度保留有效贡献输入数量。MAA不增加监督信号、参数、隐层维度或显式计数特征,且在p=1时退化为标准注意力。在四个连续时间动态图模型和三个数据集上,MAA在12个模型-数据组合中提升了11次未来链接AUC。相同隐藏表示下的线性恢复率平均提升4.49%,经家族校正后所有12组偏好连接恢复均改善。在标记时序点过程、时序知识图谱、检索增强生成及时空点过程等场景中也观察到一致效果。信息可访问性与任务效用仍分离:在MTPP中负对数似然下降,TKG中排序基本保持不变,RAG中额外信息未提升诊断头性能,下游层归一化可能抹除STPP中的信号。这些结果表明,MAA是一种通用归一化原则,可通过控制标准注意力中的重复不变性,提升面向预测的表示信息量。
原文摘要 · Abstract (English)
High task performance does not show whether a model retains prediction-relevant structural information in its internal representation. Temporal graph models, for example, can achieve high future-link AUC while basic graph statistics remain difficult to recover from the same representation. We identify one source of this gap in the weighted averaging used by standard attention: when an evidence pattern is repeated, the numerator and denominator grow at the same rate, so inputs with different amounts of accumulated evidence can produce the same aggregate. We propose Mass-Aware Attention (MAA), which generalizes standard L1 normalization to an Lp family. Under repetition, MAA makes the numerator and denominator scale at different rates, retaining the effective number of contributing inputs in the representation magnitude. It adds no supervision, parameters, hidden dimensions, or explicit count features, and recovers standard attention at p=1. Across four continuous-time dynamic graph models and three datasets, MAA improves future-link AUC in 11 of 12 model-dataset cells. Linear recovery from the same hidden representation increases by 4.49% on average, and preferential-attachment recovery improves in all 12 cells after family-wise correction. We also observe consistent evidence in marked temporal point processes, temporal knowledge graphs, retrieval-augmented generation, and spatio-temporal point processes. Information accessibility and task utility remain distinct: NLL improves in MTPP, ranking is largely preserved in TKG, additional information in RAG does not improve the diagnostic head, and downstream LayerNorm can erase the signal in STPP. These results position MAA as a general normalization principle for improving predictor-facing representation informativeness by controlling repetition invariance in standard attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。