将RNN状态与注意力机制融合，提升大模型上下文连贯性。
WuNeng: Hybrid State with Attention
- 用RWKV-7状态驱动新注意力头，增强表示能力。
- 跨头交互实现标准头与状态头动态协同，信息融合更鲁棒。
- 仅增少量参数，显著提升复杂推理与生成性能。
WuNeng架构通过融合基于RNN的RWKV-7与先进注意力机制，提升大语言模型的表达力与能力，重点强化上下文连贯性而非压缩键值缓存。在Hymba的混合头基础上，WuNeng在标准多头注意力外引入由RWKV-7状态驱动的新头，而非替换原有结构，以丰富模型表征能力。采用跨头交互技术，通过拼接、加性调制和门控融合，促进标准头、状态驱动头与新增中间头之间的动态协同，实现强健的信息整合。此外，多标记状态处理机制利用连续的RWKV-7状态捕捉全局序列依赖，显著增强表达力。这些改进仅带来极少额外参数，兼顾效率与性能，使模型在复杂推理与序列生成任务中表现卓越。WuNeng为现代神经网络架构在表达力与计算效率间取得新平衡树立了标杆。
原文摘要 · Abstract (English)
The WuNeng architecture introduces a novel approach to enhancing the expressivity and power of large language models by integrating recurrent neural network (RNN)-based RWKV-7 with advanced attention mechanisms, prioritizing heightened contextual coherence over reducing KV cache size. Building upon the hybrid-head concept from Hymba, WuNeng augments standard multi-head attention with additional RWKV-7 state-driven heads, rather than replacing existing heads, to enrich the model's representational capacity. A cross-head interaction technique fosters dynamic synergy among standard, state-driven, and newly introduced middle heads, leveraging concatenation, additive modulation, and gated fusion for robust information integration. Furthermore, a multi-token state processing mechanism harnesses the continuous RWKV-7 state to capture intricate, sequence-wide dependencies, significantly boosting expressivity. Remarkably, these enhancements are achieved with minimal additional parameters, ensuring efficiency while empowering the model to excel in complex reasoning and sequence generation tasks. WuNeng sets a new standard for balancing expressivity and computational efficiency in modern neural architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。