揭示Transformer分层特征解耦机制，解释其多任务泛化能力
Transformers as Multi-task Learners: Decoupling Features in Hidden Markov Models
- 低层提取局部上下文特征，高层实现时间解耦
- 理论构建与实验观察一致，证明模型表达能力强
- 适合研究模型可解释性与序列建模的读者
基于Transformer的模型在多种序列任务中表现出色，常通过输入输出样例实现良好性能。尽管实证成功显著，其内在机理仍缺乏系统理论理解。本文以典型的序列模型——隐马尔可夫模型（HMM）为研究对象，探究Transformer各层行为。发现：低层聚焦于特征提取，受邻近词元影响较大；高层特征逐渐解耦，呈现高度时间解耦性。基于此，我们提供了理论分析，揭示Transformer的表达能力。显式构造结果与实验观察高度吻合，为Transformer在多样化序列任务中的有效性与高效性提供了理论支持。
原文摘要 · Abstract (English)
Transformer based models have shown remarkable capabilities in sequence learning across a wide range of tasks, often performing well on specific task by leveraging input-output examples. Despite their empirical success, a comprehensive theoretical understanding of this phenomenon remains limited. In this work, we investigate the layerwise behavior of Transformers to uncover the mechanisms underlying their multi-task generalization ability. Taking explorations on a typical sequence model, i.e, Hidden Markov Models, which are fundamental to many language tasks, we observe that: first, lower layers of Transformers focus on extracting feature representations, primarily influenced by neighboring tokens; second, on the upper layers, features become decoupled, exhibiting a high degree of time disentanglement. Building on these empirical insights, we provide theoretical analysis for the expressiveness power of Transformers. Our explicit constructions align closely with empirical observations, providing theoretical support for the Transformer's effectiveness and efficiency on sequence learning across diverse tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。