从傅里叶谱角度揭示Transformer泛化机制，解释思维链为何提升性能
A Sharper Picture of Generalization in Transformers
- 基于PAC-Bayes理论，通过低阶频谱稀疏性构建平坦极小值
- 证明在上下文长度内任意稀疏布尔函数均可实现良好泛化
- 理论可解释思维链增强高阶函数泛化，适合关注模型机理的研究者
我们从目标函数傅里叶谱的角度研究Transformer在布尔域上的泛化行为。与以往基于Rademacher复杂度的分析不同，本文采用PAC-Bayes理论，证明当频谱集中在低阶成分时，可构造出低尖锐性的模型并获得非平凡的泛化界。核心思想是：对于上下文长度以内稀疏度不超过该长度的任意布尔函数，存在能实现其计算的平坦极小值；进而对理想化的低尖锐学习器应用PAC-Bayes界，得到非平凡泛化误差上界。该框架可形式化解释思维链（chain-of-thought）为何有助于高阶目标函数的泛化，并表明复杂度参数可通过属性测试高效估计。我们在实证上验证了预测结果，并通过机制可解释性研究支持理论构造在真实Transformer中的现实可行性。
原文摘要 · Abstract (English)
We study transformers' generalization behavior on boolean domains from the perspective of the Fourier spectra of their target functions. In contrast to prior work (Edelman et al., 2022; Trauger & Tosh, 2024), which derived generalization bounds from Rademacher complexity, we investigate the feasibility of obtaining generalization bounds via PAC-Bayes theory. We show that sparse spectra concentrated on low-degree components enable low-sharpness constructions with good generalization properties. Our idea is to show the existence of flat minima implementing any boolean function of sparsity no greater than the context length, and then apply a PAC-Bayes bound to an idealized low-sharpness learner, resulting in a non-vacuous generalization bound. We use this to give a formal account of why chain-of-thought improves generalization for high-degree target functions, and show that the complexity parameters in our bound can be efficiently estimated via property testing. We evaluate predictions empirically and conduct a mechanistic interpretability study to support the realism of our theoretical construction in real transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。