提出可解释的稀疏线性注意力机制,提升视觉模型效率与透明度。
Interpretable and Sparse Linear Attention with Decoupled Membership-Subspace Modeling via MCR2 Objective
- 解耦成员矩阵与子空间关系,优化MCR2目标函数
- 在ImageNet-1K上比ToST高1.08%-1.45%准确率
- 适合需要高效可解释模型的视觉任务
基于结构化表征学习的MCR2驱动白盒Transformer,统一了可解释性与效率,为视觉建模提供可靠白盒解决方案。然而,现有设计中“成员矩阵”与“子空间矩阵U”紧密耦合,在错误标记投影下导致冗余编码。为此,我们解耦MCR2目标中“成员矩阵”与“子空间U”的功能关系,从优化目标的梯度反向传播中推导出可解释的稀疏线性注意力算子。具体地,直接从输入学习成员矩阵,并从中推导稀疏子空间。梯度反向传播后得到的可解释稀疏线性注意力算子为:解耦成员-子空间注意力(DMSA)。在视觉任务上的实验表明,仅将令牌统计变换器(ToST)中的注意力模块替换为DMSA(记为DMST),不仅实现更快的编码速率下降,还在ImageNet-1K数据集上以1.08%-1.45%的顶1准确率超越ToST。相比原生Transformer架构,DMST展现出显著更高的计算效率与可解释性。
原文摘要 · Abstract (English)
Maximal Coding Rate Reduction (MCR2)-driven white-box transformer, grounded in structured representation learning, unifies interpretability and efficiency, providing a reliable white-box solution for visual modeling. However, in existing designs, tight coupling between "membership matrix" and "subspace matrix U" in MCR2 causes redundant coding under incorrect token projection. To this end, we decouple the functional relationship between the "membership matrix" and "subspaces U" in the MCR2 objective and derive an interpretable sparse linear attention operator from unrolled gradient descent of the optimized objective. Specifically, we propose to directly learn the membership matrix from inputs and subsequently derive sparse subspaces from the fullspace S. Consequently, gradient unrolling of the optimized MCR2 objective yields an interpretable sparse linear attention operator: Decoupled Membership-Subspace Attention (DMSA). Experimental results on visual tasks show that simply replacing the attention module in Token Statistics Transformer (ToST) with DMSA (we refer to as DMST) not only achieves a faster coding reduction rate but also outperforms ToST by 1.08%-1.45% in top-1 accuracy on the ImageNet-1K dataset. Compared with vanilla Transformer architectures, DMST exhibits significantly higher computational efficiency and interpretability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。