改进RWKV模型，让长文本生成更连贯准确。
Enhancing RWKV-based Language Models for Long-Sequence Text Generation
- 引入位置感知卷积移位与神经门控路由，动态调节信息流动。
- ROUGE-L提升96.5%，推理延迟仅增加2.95毫秒。
- 适合需要长序列建模的生成任务，如文档续写、对话系统。
本文提出一种增强型RWKV架构，通过自适应时间门控机制提升长上下文语言建模能力。主要创新包括：(1) 位置感知卷积移位算子，捕捉局部句法模式并保持全局连贯性；(2) 神经门控信息路由机制，动态调控词元间信息流。在文本生成任务上的全面实验表明，该模型相比基线RWKV在ROUGE-L上提升96.5%，推理延迟仅增加2.95毫秒。消融实验证实各组件独立贡献，语言学分析显示模型能自适应关注句法边界与实体一致性。所提改进在保持线性计算复杂度的同时显著增强上下文建模能力，为循环式架构在长文本生成中确立新SOTA。
原文摘要 · Abstract (English)
This paper introduces an enhanced RWKV architecture with adaptive temporal gating mechanisms for improved long-context language modeling. We propose two principal innovations: (1) a position-aware convolutional shift operator that captures local syntactic patterns while preserving global coherence, and (2) a neurally-gated information routing mechanism that dynamically regulates inter-token information flow. Through comprehensive experiments on text generation tasks, our enhanced model demonstrates superior performance compared to the baseline RWKV, achieving 96.5 relative improvement in ROUGE-L scores with only 2.95 increased inference latency. Ablation studies validate the individual contributions of each component, while linguistic analysis reveals the model's adaptive attention to syntactic boundaries and entity coherence. The proposed modifications maintain RWKV's linear computational complexity while significantly enhancing its contextual modeling capabilities, establishing new state-of-the-art performance for recurrent-style architectures in long-form text generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。