揭示混合模型中注意力与循环状态的分工：一个查内容，一个定风格。
What Attention Recalls and Recurrence Controls in Hybrid Language Models
- 通过拆分缓存和状态交换实验，分离注意力与循环状态功能。
- 注意力负责精确回忆（保留64%-98%准确率），循环状态决定语言风格（保留70%-80%）。
- 循环状态能生成语义相关但未出现的词，适合需要风格控制的任务。
混合语言模型结合注意力机制与固定大小的循环状态，但两者作用尚不明确。本文引入两种缓存层级干预：拆分预填充仅保留键值缓存或仅保留循环状态进行生成；状态交换在单次前向传播中将不同上下文的键值缓存与循环状态配对。在Qwen3.5和Falcon-H1上，两通道功能分化明显：精确信息检索仅通过注意力维持（准确率64%-98%），循环状态则完全失效；而语言风格与角色特征在循环状态中保留70%-80%与3-5倍优势，仅依赖键值缓存时降至约1%。状态交换实验证实因果关系：输出内容来自键值侧，语言风格来自循环侧。仅使用循环状态生成时，模型可产出未在上下文中出现但语义或构词相关的词。注意力提供对已述内容的查找，循环状态则决定模型接下来如何表达。
原文摘要 · Abstract (English)
Hybrid language models combine attention with a fixed-size recurrent state, but the role of each channel remains unclear. We introduce two cache-level interventions. Split-prefill keeps only the KV cache or only the recurrent state from a prefilled context, then generates an answer. State-swap pairs the KV cache from one context with the recurrent state from another in a single forward pass. On Qwen3.5 and Falcon-H1, the two channels split sharply by function. Exact retrieval survives only through attention (64-98% of full accuracy) and collapses to zero through recurrence. Output language and persona reverse the pattern: both survive recurrence (70-80% and 3-5x) while KV-only drops to ~1% language accuracy. State-swap confirms this causally: the answer takes its value from the KV side and its language from the recurrent side. Recurrent-only generation also accepts words that were never in the context but share meaning or parts with seen items. Attention provides a lookup over what was said; the recurrent state shapes how the model says it next.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。