提出用零向量令牌实现精确的注意力动态控制，提升模型对缺失信息的处理能力。
From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics
- 引入存在系数控制令牌参与度，实现零向量令牌的精确消融
- 在18组数据中使均方根误差提升0.088%~0.177%，验证有效性
- 适合需要精确处理缺失值和零输入的工业级模型改造场景
注意力掩码仅控制查询-源对的交互关系，不提供非参与令牌的表征状态。本文为每个令牌隐藏状态 $h_i$ 定义活跃存在系数 $p_i= orm{h_i}^2/(τ+ orm{h_i}^2)$，该系数同时用于门控输出与加权源贡献。提出的OAttention实现了零向量令牌的零元素性质，支持精确的空接收、空源插入及自注意力插入。基于同一存在系数，构建了局部组件（OFFN、ONorm、OInject）、带存在加权的标准归一化、以及满足 $M(H\oplus0)=M(H)\oplus0$ 的O-Closure定律。通过合约测试与GPU评估验证其正确性。在克隆的TabPFN v3回归器上，无需微调，校准后的隐状态OAttention与Full-O版本在18组匹配数据-种子组合下分别带来+0.088%和+0.177%的均方根误差提升。两块式消融实验表明，仅使用OAttention无法在普通组件中保持零状态，而完整的OTransformer路径可以。这些测试验证了精确性、激活路径兼容性与组合必要性，但不证明普遍无损、任意宿主闭包或对原点的吸引学习等通用语义。
原文摘要 · Abstract (English)
Attention masks are relation-level controls: they specify which query--source pairs may interact. They do not provide a representation-carried token state that is non-participating at the attention boundary. We assign each token hidden carrier \(h_i\) an active-presence coefficient \(p_i=\lVert h_i\rVert^2/(τ+\lVert h_i\rVert^2)\). The same coefficient has two roles: it gates information emitted by token \(i\), and it determines the mass with which token \(i\) enters computations shared with other tokens. OAttention is the support-coupled attention realization of this rule. It gates the receiver output by \(p_i\) and weights source \(j\) by \(p_j\) in both the attention numerator and partition, while retaining the standard score, visibility relation, exponential competition, and value aggregation. This makes the zero-vector token a zero element and yields exact null-receiver, null-source insertion, self-attention insertion, and empty-support properties. The same token-level presence gives local O-components (OFFN, ONorm, and OInject), presence-weighted OStandardize, the O-Closure law \(M(H\oplus0)=M(H)\oplus0\), and an OTransformer by residual and compositional closure. The canonical operator is checked by contract tests and a GPU evaluation. In a zero-fine-tuning retrofit of a cloned pretrained TabPFN v3 regressor, calibrated hidden-carrier OAttention and Full-O variants change mean RMSE by $+0.088\%$ and $+0.177\%$, respectively, over 18 matched dataset--seed cases. A two-block ablation shows that OAttention alone does not preserve a NULL state through ordinary host components, whereas the OTransformer path does. These are scoped tests of exactness, active-path compatibility, and compositional necessity; they do not establish universal no-loss, arbitrary-host closure, learned attraction to the origin, or a general semantics for missing values.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。