arXiv:2601.19221cs.CL2026-01

让RNN状态可编辑,用扩散模型动态调整语言模型参数。

DREAMSTATE: Diffusing States and Parameters for Recurrent Large Language Models

  • 用扩散Transformer建模RNN状态,实现状态生成与编辑。
  • 新架构使固定循环变为上下文感知的动态函数,训练稳定。
  • 适合研究RNN可解释性与动态模型设计的读者。

现代循环神经网络(如RWKV)凭借强大的局部建模能力和固定大小的状态,相比标准Transformer具有显著优势。然而,对其内部状态作为可编辑知识表示的研究仍十分匮乏。为此,我们提出DREAMSTATE框架,利用条件扩散Transformer(DiT)直接建模状态的概率流形,实现状态的生成与编辑。通过t-SNE可视化和受控生成实验验证了该表示的结构性。在此基础上,我们进一步设计一种新型混合架构,将RNN的局部优势与全局上下文适应性结合:引入并行的DiT模块,处理可变长度全局上下文,动态生成并调整核心循环模块的WKV参数,将固定循环机制转化为上下文感知的动态函数。实验表明,该混合模型可通过多目标损失稳定训练,验证了设计可行性。本工作不仅开辟了RNN状态表示的新方向,也为未来模型设计提供了具体参考。代码已公开于https://huggingface.co/2dgx41s/DreamState。

原文摘要 · Abstract (English)

Modern Recurrent Neural Networks (RNNs), such as RWKV, are distinguished by their powerful short-range modeling capabilities and efficient fixed-size states, which constitute a core advantage over standard Transformers. However, there is a significant lack of research into their internal state as an editable knowledge representation. To fill this gap, we first explore the representational properties of the RWKV state by proposing the DREAMSTATE framework. This framework utilizes a conditional Diffusion Transformer (DiT) to directly model the probability manifold of the state, enabling its generation and editing. The structural nature of this representation is validated through t-SNE visualizations and controlled generation experiments. After successfully uncovering and modeling the state's representational potential, we further propose a novel hybrid architecture that combines the local advantages of RNNs with global context adaptability. This architecture features a parallel DiT that processes a variable-length global context to dynamically generate and adjust the core recurrent module's WKV parameters, transforming the fixed recurrence mechanism into a context-aware dynamic function. Experiments demonstrate that this hybrid model can be trained stably via a multi-objective loss, validating its design feasibility. Our work not only opens a new research direction for RNN state representation but also provides a concrete architectural reference for future model design. The code is publicly available at: https://huggingface.co/2dgx41s/DreamState.

RNN扩散模型动态参数状态编辑

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。