用简单平均探明序列模型的结构与内容边界，发现固定累积会丢失关键信息。
EMA Is Not All You Need: Mapping the Boundary Between Structure and Content in Recurrent Context
- 用EMA作为无门控的递归上下文，测试固定系数积累的表达能力
- 多时标EMA在语法角色任务上达监督BiGRU的96%，且零标签下更优
- 信息压缩不可逆，只有自适应选择才能恢复被丢弃的内容
高效序列模型相比简单时间平均究竟获得了什么？我们使用指数移动平均（EMA）迹线——最简单的递归上下文（无门控、无内容检索）——作为可控探针，来刻画固定系数累积所能表示与不能表示的边界。EMA迹线编码时间结构：具有多时标迹线的赫布架构在语法角色标注任务上达到监督BiGRU的96%性能，且在无标签条件下优于监督模型。但EMA迹线破坏了词元身份：一个仅使用EMA上下文的1.3亿参数语言模型在C4数据集上困惑度为260（是GPT-2的8倍），而将线性预测器替换为完整softmax注意力后损失不变，表明全部差距都源于迹线本身。这些迹线实施有损、数据无关的压缩；根据数据处理不等式，下游预测器无法恢复被丢弃的信息。固定系数累积（无论时间或深度方向）会导致不可逆的信息稀释，唯有学习的、输入相关的筛选机制方可解决。
原文摘要 · Abstract (English)
What exactly do efficient sequence models gain over simple temporal averaging? We use exponential moving average (EMA) traces, the simplest recurrent context (no gating, no content-based retrieval), as a controlled probe to map the boundary between what fixed-coefficient accumulation can and cannot represent. EMA traces encode temporal structure: a Hebbian architecture with multi-timescale traces achieves 96% of a supervised BiGRU on grammatical role assignment with zero labels, surpassing the supervised model on structure-dependent roles. EMA traces destroy token identity: a 130M-parameter language model using only EMA context reaches C4 perplexity 260 (8x GPT-2), and a predictor ablation (replacing the linear predictor with full softmax attention) yields identical loss, localizing the entire gap to the traces. The traces apply lossy, data-independent compression; by the data processing inequality, no downstream predictor can recover the discarded information. Fixed-coefficient accumulation, whether across time or depth, suffers irreversible information dilution that only learned, input-dependent selection can resolve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。