用印尼数学教学法训练小模型,实现高效算术推理。
Arithmetic Pedagogy for Language Models

- 基于左到右计算流程构建思维链监督数据
- 86M小模型达80%以上准确率,超越更大模型
- 适合想用小模型做算术推理的研究者
我们探究人类数学教学法是否可指导语言模型的算术推理训练。基于印尼的GASING教学法——一种按词元生成因果顺序从左到右解决基础算术的方法——我们将每个运算操作转化为可序列化的计算过程,并将其执行轨迹转化为自然语言思维链(CoT)监督信号。使用仅8600万参数的小型GPT-2解码器,搭配印尼语音节黏着式TOBA分词器,从零开始训练,仅采用下一个词预测目标,无强化学习或奖励优化。训练过程中观察到三个明显阶段。通过注意力掩码干预、残差流探查和逻辑透镜分析发现:模型先内化程序路径,随后发展出无需显式步骤计算即可检索中间结果的“心算”能力。该模型在保留问题上准确率超80%,表现优于远大于它的语言模型,表明基于教学法的针对性训练可在小规模下实现强大且经济的算术能力。
原文摘要 · Abstract (English)
We investigate whether methods of human mathematics pedagogy can guide the training of language models toward arithmetic reasoning. Building on the GASING method -- an Indonesian pedagogy that solves basic arithmetic through a left-to-right procedure aligned with the causal order of token generation -- we operationalize each operation as a computational procedure whose execution trace is serialized into natural-language Chain-of-Thought (CoT) supervision. A small GPT-2 decoder (86M parameters) with a syllabic-agglutinative TOBA tokenizer for Indonesian is trained from scratch on this data using only a next-token prediction objective, without reinforcement learning or reward-based optimization. Monitoring training reveals three distinct learning phases, and mechanistic analyses -- attention-masking interventions on the CoT information graph, residual-stream probing, and logit-lens inspection -- show that the model first internalizes a procedural pathway and subsequently develops an associative, ``mental-arithmetic'' capacity that retrieves intermediate results without explicit step-by-step computation. The trained model reaches over 80% accuracy on held-out problems and attains competitive performance against substantially larger language models, indicating that targeted, pedagogically grounded training can yield strong and economical arithmetic capability at small scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。