Transformer学乘法时其实是在离散对数空间里做加法,这揭示了隐藏的可解释结构。
The Discrete-Log Clock: How a Transformer Learns Modular Multiplication

- 用乘法特征变换替代传统傅里叶变换,发现乘法学习本质是稀疏频谱
- 仅4个关键频率承载主要能量,96.9%的MLP神经元只响应单一频率
- 通过离散对数重排神经元激活图,显现出二维周期性结构
当小型Transformer掌握模乘法运算时,以往研究发现其学习到的嵌入具有‘稠密’傅里叶频谱,需所有频率成分。这与模加法形成对比——后者只需少量关键频率即可。本文指出,这种密度是因分析基选择不当所致。模乘法的自然傅里叶变换应为乘法特征变换,它将定义在乘法群$(\mathbb{Z}/p\mathbb{Z})^*$上的函数分解为其不可约表示。将该变换应用于在$ a \cdot b \bmod 113 $上训练的模型,发现嵌入频谱变得高度稀疏(吉尼系数0.58,相较加法基的0.07),仅4个关键频率携带显著能量。此外,96.9%的MLP神经元对单一乘法频率有清晰响应,神经元激活热图在按离散对数重排后呈现2D周期性结构。结果表明,Transformer将模乘法转化为离散对数空间中的加法,实现类似Nanda等人提出的‘钟表算法’。该方法具通用性:匹配任务代数结构的分析基,能揭示标准工具视为噪声的可解释结构。
原文摘要 · Abstract (English)
When small transformers grok modular multiplication, prior work reports that the learned embedding has a "dense" Fourier spectrum requiring all frequencies. This contrasts with modular addition, where only a sparse set of key frequencies suffices. We show this density is an artifact of analyzing in the wrong basis. The natural Fourier transform for multiplication is not the standard additive DFT but the multiplicative character transform, which decomposes functions on the multiplicative group $(\mathbb{Z}/p\mathbb{Z})^*$ into its irreducible representations. Applying this transform to a grokked transformer trained on $a \cdot b \bmod 113$, we find the embedding spectrum becomes highly sparse (Gini coefficient 0.58 vs. 0.07 in the additive basis) with only 4 key frequencies carrying significant energy. Furthermore, 96.9% of MLP neurons are cleanly tuned to a single multiplicative frequency, and neuron activation heatmaps reveal 2D-periodic structure when reordered by the discrete logarithm. These results demonstrate the transformer reduces multiplication to addition in discrete-log space, implementing a "Discrete-Log Clock" algorithm analogous to Nanda et al.'s Clock algorithm for addition. The methodology generalizes: matching the analysis basis to the algebraic structure of the task reveals interpretable structure where standard tools see noise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。