让Transformer内部自动学会符号计算,像搭积木一样智能调用工具。
Internalizing Tools as Morphisms in Graded Transformers
- 用分级向量空间和类型映射实现符号操作的内化
- 通过可微路由选择激活特定符号变换,提升效率与可解释性
- 适合研究神经符号系统、模型可解释性的学者参考
我们提出一种针对Transformer的分级符号计算内化框架。隐藏空间被赋予分级结构 $V=\bigoplus_{g\in G}V_g$,符号操作以带类型的块映射(态射)$ϕ_{h\leftarrow g}:V_g\to V_h$ 形式实现,并由可微路由策略选择性激活。一个自监督的分级效用函数(定义为候选态射带来的损失减少)控制激活,实现稀疏且可解释的行为。我们建立了代数与几何基础:一个对象为同调分量、态射为允许的等级转移的内部模型范畴;伴随对编码类型往返路径;以及基于KL增益、Bregman散度镜面下降和费雪自然梯度的信息几何解释。方法上,设计了效用感知路由机制与目标函数,保持全程可微。分析案例与轻量级验证展示了在混合符号-语言任务中选择性态射激活的有效性。该框架统一了符号计算、几何结构与自监督学习于分级Transformer形式下,同时将先前外部工具范式(如Toolformer)作为函子内化的特例纳入其中。
原文摘要 · Abstract (English)
We introduce a graded formulation of internal symbolic computation for transformers. The hidden space is endowed with a grading $V=\bigoplus_{g\in G}V_g$, and symbolic operations are realized as typed block maps (morphisms) $ϕ_{h\leftarrow g}:V_g\to V_h$ that are activated selectively by a differentiable routing policy. A self-supervised \emph{graded utility functional}, defined as the loss reduction induced by a candidate morphism, governs activation and yields sparse, interpretable behavior. We develop the algebraic and geometric foundations: an internal model category whose objects are homogeneous components and whose morphisms are admissible grade transitions; adjoint pairs encoding typed round trips; and information-geometric interpretations in terms of KL gain, mirror descent with Bregman divergences, and Fisher natural gradients. Methodologically, we specify a utility--aware routing mechanism and objective that remain fully end-to-end differentiable. Analytic case studies and lightweight sanity checks illustrate selective morphic activation on hybrid symbolic-linguistic tasks. The framework unifies symbolic computation, geometry, and self--supervised learning within the \emph{graded transformer} formalism \cite{sh-89,sh-95}, while subsuming prior external-tool paradigms (e.g., Toolformer \cite{toolformer2023}) as a special case via functorial internalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。