将扩散模型与RWKV结合，实现高效并行文本生成。
Triplet-Block Diffusion RWKV
- 用三元组块结构统一因果模型与扩散模型的架构矛盾。
- 72亿参数模型在8项任务上精度相当，推理速度提升1.6倍。
- 适合追求高速文本生成且对延迟敏感的应用场景。
因果Transformer语言模型存在严格串行解码和每步注意力开销为二次方的问题。虽然线性时间因果模型与离散扩散模型各自解决了这些缺陷，但两者的融合仍存在根本矛盾：扩散模型需要双向注意力，而因果模型是单向的。为此，我们提出B^3D-RWKV，一种将模型O(L)推理效率与并行、双向离散扩散相结合的RWKV变体，通过三元组块布局方法实现统一。B^3D-RWKV-7.2B在8项任务套件上达到与现有模型相当的准确率，同时在解码吞吐量上显著优于基线，平均提速1.6倍。
原文摘要 · Abstract (English)
Causal Transformer language models suffer from strictly sequential decoding and a quadratic per-step attention cost. While linear-time causal models and discrete diffusion models each address these weaknesses, their integration remains inherently inconsistent: diffusion requires bidirectional attention, while causal models are unidirectional. To unify these architectures, we propose $B^3D-RWKV$, a diffusion RWKV variant that integrates the model's $O(L)$ inference efficiency with parallel, bidirectional discrete-diffusion through a \emph{triplet-block layout} method. $B^3D-RWKV-7.2B$ reaches comparable accuracy on an 8-task suite versus existing models while significantly outperforming baselines in decoding throughput with an average of $\mathbf{1.6\times}$ speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。