用跨注意力提升RWKV-7文本图像生成能力，实现高效多模态对齐。
Cross-attention for State-based model RWKV-7
- 在RWKV-7中引入跨注意力机制CrossWKV，单步融合文本与图像信息。
- 在ImageNet上达成FID 2.88、CLIP得分0.33，性能达顶尖水平。
- 支持高分辨率生成与动态状态控制，适合复杂多模态任务。
我们提出CrossWKV，一种针对基于状态的RWKV-7模型的新颖跨注意力机制，旨在增强文本到图像生成的表达能力。利用RWKV-7的线性复杂度加权键值（WKV）架构，CrossWKV在单次传递中整合文本与图像模态，采用向量门控和低秩适应（LoRA）的广义δ规则，实现更优的跨模态对齐。与Transformer模型不同，CrossWKV的非对角、输入依赖的转移矩阵使其能表示超越TC⁰复杂度类的复杂函数，包括所有正则语言，其在S₅置换建模等状态追踪任务中表现优异。在LAION-5B和ImageNet数据集上的扩散模型框架DIR-7中评估，该方法在ImageNet 256x256上取得FID 2.88和CLIP分数0.33，达到当前最优水平，同时具备强鲁棒性与多样化提示的泛化能力。模型的增强表达力，结合恒定内存使用和线性扩展特性，使其成为先进跨模态任务的强大解决方案，有望应用于高分辨率生成与动态状态操作。代码见https://github.com/TorchRWKV/flash-linear-attention。
原文摘要 · Abstract (English)
We introduce CrossWKV, a novel cross-attention mechanism for the state-based RWKV-7 model, designed to enhance the expressive power of text-to-image generation. Leveraging RWKV-7's linear-complexity Weighted Key-Value (WKV) architecture, CrossWKV integrates text and image modalities in a single pass, utilizing a generalized delta rule with vector-valued gating and low-rank adaptations (LoRA) to achieve superior cross-modal alignment. Unlike Transformer-based models, CrossWKV's non-diagonal, input-dependent transition matrix enables it to represent complex functions beyond the $\mathrm{TC}^0$ complexity class, including all regular languages, as demonstrated by its ability to perform state-tracking tasks like $S_5$ permutation modeling. Evaluated within the Diffusion in RWKV-7 (DIR-7) on datasets such as LAION-5B and ImageNet, CrossWKV achieves a Frechet Inception Distance (FID) of 2.88 and a CLIP score of 0.33 on ImageNet 256x256, matching state-of-the-art performance while offering robust generalization across diverse prompts. The model's enhanced expressivity, combined with constant memory usage and linear scaling, positions it as a powerful solution for advanced cross-modal tasks, with potential applications in high-resolution generation and dynamic state manipulation.Code at https://github.com/TorchRWKV/flash-linear-attention
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。