循环变压器中全局工作空间仍存在,但信息传递方式因循环结构改变。
Looped Transformers under the Jacobian Lens: Does the Global Workspace Survive Recurrence?

- 用虚拟展开适配器扩展雅可比视角,分析循环架构中的表征机制。
- Ouro在每轮循环中重建工作空间内容,信息无法跨轮传递;Huginn则能跨16轮传递。
- 适合研究循环神经网络表征、注意力机制与可解释性的人阅读。
近期研究发现标准前馈Transformer中存在一个中深度的可表述、因果有效的表征带——功能上类似全局工作空间。当深度通过循环而非多层堆叠实现时,这种功能是否仍存在尚不明确。环状和深度循环Transformer为该问题提供了直接检验:它们在深度方向复用相同权重。本文通过虚拟展开适配器将雅可比视角扩展至迭代架构,并对Ouro-2.6B(48层循环4次,深度监督)和Huginn-0125(4层核心循环16次,训练用于潜在推理)应用完整的工作空间评估套件——包括镜头拟合、读出及十一组因果实验,以Qwen3.6-27B(64个未绑定层)为基准。结果表明,两模型的迭代部分均形成工作空间,但访问方式受循环影响:Ouro在每轮循环中重建内容,线性传输无法跨越循环边界,写入与消融必须覆盖所有剩余循环;而Huginn能将内容跨全部16次循环传递,读取、写入与消融仅作用于约两个循环的滑动窗口内。新注入内容能否被表述取决于每轮显式监督,而已有内容能否被引导则不受此影响。
原文摘要 · Abstract (English)
Recent work identifies a mid-depth band of verbalisable, causally potent representations in a standard feedforward transformer --- a functional analogue of a global workspace. Whether the same workspace functionality emerges when depth is implemented through recurrence rather than a stack of distinct layers remains unknown. Looped and depth-recurrent transformers provide a direct test of this question because they reuse the same weights across depth. We extend the Jacobian lens to iterated architectures using a virtual-unrolling adapter. We apply the full workspace suite --- lens fitting, readout, and eleven causal experiment families --- to Ouro-2.6B (48 layers looped 4 times, deeply supervised) and Huginn-0125 (a 4-layer core recurred 16 times, trained for latent reasoning), using Qwen3.6-27B (64 untied layers) as the standard baseline. We find that a workspace forms in the iterated part of each architecture, but that recurrence changes how it can be accessed. Ouro reconstructs workspace content in every loop, and linear transport cannot carry that content across loop boundaries; writes and ablations must therefore span every remaining loop. Huginn carries content forward across all sixteen recurrences, while reads, writes, and ablations act only within a sliding window of roughly two recurrences. Whether newly injected content can be verbalised tracks explicit per-iteration supervision; whether existing content can be steered does not.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。