让递归变压器每步动态调整权重，提升深层推理能力。
Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned LoRA Modulation
- 用控制器根据输入状态生成每步的调制向量，实现递归步骤的输入依赖性。
- 在17层递归架构上，训练损失降低43.4%，恢复51.3%的性能损失。
- 仅增加920万参数，优于同等规模静态LoRA，在多深度和多秩下持续领先。
递归变压器通过复用共享权重块来节省参数，但每步执行相同变换，限制了操作组合能力。我们提出Ouroboros，为递归块附加一个轻量级控制器超网络。控制器观察当前隐藏状态，生成每步的对角调制向量，并作用于冻结的SVD初始化LoRA基，使每一步的变换随输入变化。结合门控递归（初始保留率88%）和每步归一化，实现稳定深层迭代。在将Qwen2.5-3B拆分为Prelude/Recurrent/Coda架构（保留17/36层）的实验中，Ouroboros相比未修改的17层基线，训练损失降低43.4%，恢复了因层移除导致51.3%的性能差距。整个系统仅增加920万可训练参数（控制器、门控与每步归一化），在深度1时优于等规模静态每步LoRA 1.44个损失点，且在所有测试深度（1,4,8,16）和秩（8,32,64）上保持领先。我们发现门控递归至关重要：无门控时，递归应用反而使模型更差。这些提升基于训练分布；在外部文本上，控制器尚未超越基线，我们归因于下游层冻结，并详细讨论该局限性。
原文摘要 · Abstract (English)
Recursive transformers reuse a shared weight block across multiple depth steps, trading parameters for compute. A core limitation: every step applies the same transformation, preventing the model from composing distinct operations across depth. We present Ouroboros, a system that attaches a compact Controller hypernetwork to a recursive transformer block. The Controller observes the current hidden state, produces a per-step diagonal modulation vector, and applies it to frozen SVD-initialized LoRA bases, making each recurrence step input-dependent. We combine this with gated recurrence (bias-initialized to 88% retention) and per-step LayerNorm for stable deep iteration. On Qwen2.5-3B split into a Prelude/Recurrent/Coda architecture (17 of 36 layers retained), Ouroboros reduces training loss by 43.4% over the unmodified 17-layer baseline, recovering 51.3% of the performance gap caused by layer removal. The full system adds only 9.2M trainable parameters (Controller, gate, and per-step norms) yet outperforms equivalently-sized static per-step LoRA by 1.44 loss points at depth 1 and remains ahead across all tested depths (1, 4, 8, 16) and ranks (8, 32, 64). We also find that gated recurrence is essential: without it, recursive layer application makes the model strictly worse. These gains are measured on the training distribution; on held-out text, the Controller does not yet improve over the baseline, a limitation we attribute to frozen downstream layers and discuss in detail. Code: https://github.com/RightNow-AI/ouroboros
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。