共享模型通过输入注入差异实现状态分工,无需模块拆分。
One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer

- 同一Transformer在不同更新阶段注入输入,触发内部角色分化。
- 实验显示高状态保留全局规划,低状态维持局部不确定性。
- 适合研究神经网络内在机制与可解释性的人关注。
共享权重的循环Transformer能否在不划分模块的情况下发展出不同内部角色?我们通过极简的非对称输入循环(AIR)架构进行研究,其中同一Transformer被用于两种更新(L和H),唯一区别是输入编码仅在L更新时注入。在Sudoku-Extreme和Maze任务中,解码轨迹揭示稳定分化:$ζH$ 表现为完全投入的提案状态,而 $ζL$ 保持局部不确定性和动态中间结构。冻结实验表明,这种分化与状态动力学相关:在Sudoku中,冻结$ζH$会减少$ζL$的内容变化,反之则增加;在Maze中,任一状态冻结都会导致另一状态内容变化加剧。消融实验显示,为诱导分工,共享模型需能区分更新类型,可通过输入注入不对称或单独层级标记实现。机制分析表明,无论在Sudoku还是Maze中,L更新的注意力始终更局域。结果表明,在双状态循环设置中,明确的状态标识信号可在共享参数模型内诱导出稳定且相关的功能角色。
原文摘要 · Abstract (English)
Can a shared-weight recurrent Transformer develop distinct internal roles without being partitioned into separate modules? We study this in Asymmetric Input Recurrence (AIR), a minimal two-state reasoning architecture in which the same Transformer model is reused for both updates (per literature, L and H) and the only built-in difference in the update rule is that the encoded input is injected during L-updates but not H-updates. Across Sudoku-Extreme and Maze, decoded rollouts reveal a consistent split: $\zH$ behaves like a fully committed proposal state, whereas $\zL$ retains local uncertainty and shifting intermediate structure. Freeze experiments show that this split is, in practice, related to the model's state dynamics: in Sudoku, freezing $\zH$ reduces $\zL$'s content changes whereas freezing $\zL$ increases $\zH$'s, while in Maze, freezing either state increases content changes in the other state. Ablations show that to induce specialization, the shared model needs to be able to tell the two update types apart, either from input injection asymmetry or from a separate level token. Mechanistically, attention analysis shows that L-updates are consistently more local than H-updates in both Sudoku and Maze. Together, these results show that, in a two-state recurrent setting, a clear state-identity signal can induce stable, related functional roles inside a shared-parameter recurrent Transformer. Code is available at \href{https://github.com/juchengshen/air}{\textcolor{blue}{https://github.com/juchengshen/air}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。