Llama模型用十进制加法推理循环概念，而非直接模运算。
Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
- 用十进制加法计算输入和，再映射回循环空间。
- 关键特征周期为2、5、10，非概念本身周期（如12）。
- 仅28个神经元复用，适用于所有循环推理任务。
Llama-3.1-8B在处理循环概念（如‘八月后六个月是哪个月？’）时，尽管其表示具有环形结构，却未直接进行模12等周期运算，而是采用通用的十进制加法机制：先将数字与月份相加（如六+八=14），再将结果映射回循环空间（14→二月）。该模型通过任务无关的傅里叶特征实现加法，其特征周期分别为2、5、10，符合十进制算法规律，而非概念本身的周期。我们发现仅有28个MLP神经元（约第18层MLP的0.2%）被跨任务复用，可划分为多个不重叠簇，每簇负责一个特定周期的傅里叶特征计算。本研究揭示了因果抽象与特征几何如何共同作用，深化对大模型推理机制的理解。
原文摘要 · Abstract (English)
Does structure in representations imply structure in computation? We study how Llama-3.1-8B reasons over cyclic concepts (e.g., "what month is six months after August?"). Even though Llama-3.1-8B's representations for these concepts are circularly structured, we find that instead of directly computing modular addition in the period of the cyclic concept (e.g., 12 for months), the model re-uses a generic addition mechanism across tasks that operates independently of concept-specific geometry. First, it computes the sum of its two inputs using base-10 addition (six + August=14). Then, it maps this sum back to cyclic concept space (14->February). We show that Llama-3.1-8B uses task-agnostic Fourier features to compute these sums--in fact, these features have periods that respect standard base-10 addition, e.g., 2, 5, and 10, rather than the cyclic concept period (e.g., 12 for months). Furthermore, we identify a sparse set of 28 MLP neurons re-used across all tasks (approximately 0.2% of the MLP at layer 18) that can be partitioned into disjoint clusters, each computing the sum for a Fourier feature with a different period. Our work highlights how an interplay between causal abstraction and feature geometry can deepen our mechanistic understanding of LMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。