arXiv:2601.10007cs.LGcs.CL2026-01被引 1

用连续深度ODE替换离散层,实现生成属性的实时调控。

Continuous-Depth Transformers with Learned Control Dynamics

  • 将变换器深度视为连续变量,由学习的向量场控制
  • 情感控制准确率达98%/88%,轨迹偏差仅0.068%
  • 支持推理时动态调节,适合需要可控生成的场景

我们提出一种混合变换器架构,用连续深度的神经微分方程(ODE)块替代离散中间层,通过学习的引导信号实现在推理阶段对生成属性的控制。与标准变换器固定层数不同,该方法将深度视为受学习向量场 $F_θ(H, τ, u)$ 控制的连续变量,其中 $u$ 是通过显式拼接注入的低维控制信号。我们在四个实验中验证该架构:(1)梯度流稳定,零梯度爆炸/消失事件;(2)语义引导在正/负情感控制上分别达到98%/88%准确率;(3)连续插值验证显示固定与自适应求解器间轨迹偏差仅为0.068%;(4)效率基准测试表明延迟与标准离散基线相当。此外,自适应ODE求解器揭示了学习动力学中的几何结构:控制信号将向量场划分为具有不同曲率特征的独立动力学区域。伴随法实现 $O(1)$ 内存训练,无论积分深度如何。结果表明,带有学习控制信号的连续深度动态为可调控语言生成提供了可行且高效的机制。

原文摘要 · Abstract (English)

We present a hybrid transformer architecture that replaces discrete middle layers with a continuous-depth Neural Ordinary Differential Equation (ODE) block, enabling inference-time control over generation attributes via a learned steering signal. Unlike standard transformers that process representations through fixed discrete layers, our approach treats depth as a continuous variable governed by a learned vector field $F_θ(H, τ, u)$, where $u$ is a low-dimensional control signal injected via explicit concatenation. We validate the architecture through four experiments: (1) gradient flow stability with zero exploding/vanishing gradient events, (2) semantic steering achieving 98\%/88\% accuracy for positive/negative sentiment control, (3) continuous interpolation validated by a negligible 0.068\% trajectory divergence between fixed and adaptive solvers, and (4) efficiency benchmarking demonstrating latency parity with standard discrete baselines. Additionally, we show that adaptive ODE solvers reveal geometric structure in the learned dynamics: the control signal partitions the vector field into distinct dynamical regimes with different curvature characteristics. The adjoint method enables $O(1)$ memory training regardless of integration depth. Our results demonstrate that continuous-depth dynamics with learned control signals provide a viable, efficient mechanism for steerable language generation.

连续深度可控生成ODE变换器

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。