arXiv:2605.27259cs.LG2026-05被引 3

用范畴论统一注意力、扩散与自条件模型,揭示其深层共性。

Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning

论文配图:Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning
图 1 · 摘自论文原文
  • 基于范畴论的左肯普扩展框架,统一流行Transformer结构的设计思路。
  • 在WikiText-2/103上,大宽度下二次型KET在严格因果任务中表现最优。
  • 自条件信息带来最大性能提升,远超邻域设计改进,适合理论研究者。

我们提出范畴论统一框架下的肯普扩展变换器(Kan Extension Transformers, KETs),用于描述多种Transformer架构的共性设计语言。层可视为加权结构扩展算子:注意力依赖标记邻域,几何混合使用稀疏关联,而KET采用单纯形源。当源值为函子、权重为可表示同态对象或指定预函子作用、聚合实现相应余端时,该算子才是真正的增强左肯普扩展;否则仅具‘肯普风格’的解释意义。预测-分离模块通过预测载体阻断梯度,避免教师强制隐藏状态传播,但仅分离不足以实现严格自回归:目标时刻t所消耗的载体必须可由t前缀测量。我们在宾夕法尼亚树库、WikiText-2和WikiText-103上评估了12种实现,涵盖宽度d=64,256和深度L=2,8,16的组合。在严格因果与自条件两种范式下,二次型KET在WikiText-2和WikiText-103上均优于其他严格因果架构;跨范式收益主要来自额外自条件信息,而非邻域设计本身。

原文摘要 · Abstract (English)

We propose Kan Extension Transformers (KETs) as a categorical design language for a diverse group of Transformer implementations. A layer can be viewed generally as a weighted structured extension operator: attention uses token neighborhoods, geometric mixing uses sparse incidences, and KET uses simplicial sources. This operator is an actual enriched left Kan extension only when the source values are functorial, the weights are representable hom-objects (or the specified profunctor action), and aggregation realizes the corresponding coend; otherwise ``Kan-style'' denotes an interpretation rather than an identity theorem. Predict-detach blocks gradients through a predictive carrier and avoids transporting teacher-forced hidden states, but detach alone does not make a noncausal update strictly autoregressive: every carrier consumed at target $t$ must also be measurable from the prefix available at $t$. We evaluate 12 implementations on Penn Treebank, WikiText-2, and WikiText-103 across strict-causal and self-conditioned regimes, using widths $d=64,256$ and depths $L=2,8,16$ across the reported studies. Quadratic KET is strongest among the compared strict-causal architectures on WikiText-2 and WikiText-103; the largest cross-regime gains arise from additional self-conditioning information, not neighborhood design alone.

范畴论Transformer自条件注意力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。