提出可自适应上下文的值投影机制,提升集合变换模型对复杂依赖关系的表达能力。
Matrix Zonotopic Attention: A Context-Adaptive Value Projection for Set Transformers

- 用矩阵多面体族替代固定值投影,实现输入相关的动态映射
- 实验表明在高秩稀疏组合任务上性能显著优于标准注意力
- 适合处理依赖复杂结构的集合预测问题,如几何形状生成
多头注意力将输入依赖的Softmax路由与输入无关的线性值投影结合,导致每个样本的输出映射相同。本文研究这种不对称性对置换不变集合目标的影响,提出目标算子的变换自由度(TDOF)作为复杂度度量,量化精确表示所需输入相关的方向数。深度分离分析显示,上下文固定的注意力需深度与目标TDOF成正比,而单层具有上下文自适应值族的模型即可实现相同表示。基于此,我们提出矩阵多面体注意力(MZAttn),以中心矩阵加生成矩阵加权和的形式构建上下文自适应值投影:初始时退化为标准多头注意力,保持置换等变性,并具有数据驱动的可达性解释。在多种集合预测任务上的实验验证了理论预测:该架构优势在依赖高秩稀疏组合结构的目标上明显,而在统计聚合类目标上与参数匹配的标准注意力相当。
原文摘要 · Abstract (English)
Multi-head attention combines an input-dependent softmax routing with an input-independent linear value projection, so the per-sample operator mapping aggregated values to outputs is the same for every input set. We study the consequences of this asymmetry for permutation-invariant set targets. We introduce the Transformation Degrees of Freedom (TDOF) of a target operator, a complexity measure counting the input-dependent directions an exact representation requires, and present a depth-separation analysis showing that context-rigid attention needs depth proportional to the target's TDOF, whereas a single layer with a context-adaptive value family can represent the same target. Building on this analysis, we propose Matrix Zonotopic Attention (MZAttn), which replaces the fixed value projection with a context-adaptive matrix-zonotope family: a centre matrix plus a sum of generator matrices weighted by input-dependent gates. The construction reduces to standard multi-head attention at initialisation, preserves permutation equivariance, and admits a data-driven reachability interpretation. Experiments on a range of set-prediction tasks are consistent with the TDOF prediction that the architectural advantage is selective: it appears on targets that depend on the input set in a high-rank, sparsely combinatorial way, and is small on aggregate-statistic targets where parameter-matched standard attention is already competitive.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。