arXiv:2605.05151cs.LGcs.AI2026-05被引 1

发现时间序列模型无需复杂表示也能高效预测,简单结构已足够。

Superposition Is Not Necessary: A Mechanistic Interpretability Analysis of Transformer Representations for Time Series Forecasting

  • 用稀疏自编码器分析注意力层中间特征,检验超叠加机制是否存在。
  • 扩展字典规模后性能仅下降0.214%,多数特征未被激活。
  • 关键特征干预几乎不影响预测结果,说明表征稀疏且稳定。

Transformer架构广泛应用于时间序列预测,但其在自然语言处理中成功的表征机制是否适用于时间序列数据尚不清楚。简单线性模型(如DLinear)持续表现出竞争力,却缺乏机理解释。本文通过稀疏自编码器(SAEs)分析PatchTST的内部表示,发现单层窄维Transformer在常见基准上表现与深层模型相当。在后GELU中间前馈层激活值上训练SAEs,字典规模从0.5倍到4.0倍原维度,性能变化仅为平均0.214%,大量超完备字典特征未被使用。对主导隐变量进行因果干预后,预测扰动极小。所有设置下均无证据表明前馈网络表示依赖强超叠加。表征保持稀疏、字典扩张后仍稳定,且对隐变量干预不敏感。结果表明,标准预测基准下超叠加并非高性能必要条件,暗示这些任务可能无需语言建模所需的丰富组合表征,从而解释了简单线性模型的持续竞争力。

原文摘要 · Abstract (English)

Transformer architectures have been widely adopted for time series forecasting, yet whether the representational mechanisms that make them powerful in NLP actually engage on time series data remains unexplored. The persistent competitiveness of simple linear models such as DLinear has fueled ongoing debate, but no mechanistic explanation for this phenomenon has been offered. We address this gap by applying sparse autoencoders (SAEs), a tool from mechanistic interpretability, to probe the internal representations of PatchTST. We first establish that a single-layer, narrow-dimensional transformer matches the forecasting performance of deeper configurations across commonly used benchmarks. We then train SAEs on the post-GELU intermediate FFN activations with dictionary sizes ranging from 0.5x to 4.0x the native dimensionality. Expanding the dictionary yields negligible downstream performance change (average 0.214%), with large portions of overcomplete dictionaries remaining inactive. Targeted causal interventions on dominant latent features produce minimal forecast perturbation. Across all evaluated settings, we observe no empirical evidence that the analyzed FFN representations rely on strong superposition. Instead, the representations remain sparse, stable under aggressive dictionary expansion, and largely insensitive to latent interventions. These results demonstrate that superposition is not necessary for competitive performance on standard forecasting benchmarks, suggesting they may not demand the rich compositional representations that drive transformer success in language modeling, and helping explain the persistent competitiveness of simple linear models

可解释性时间序列Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。