让小模型解数学题，用公式格式替代自然语言
Small Language Models are Equation Reasoners
- 将数学推理转为纯公式格式，降低表达歧义
- T5-Tiny模型在公式格式下准确率提升显著
- 适合资源有限但需数学推理的场景
链式思维（CoT）推理使大语言模型在多项NLP任务中表现优异，包括算术问题求解。然而，这种成功无法推广到小型语言模型（sLM），如T5，因其容量有限且缺乏大模型的涌现能力。现有通过知识蒸馏增强sLM的方法虽有一定改进，但仍面临自然语言表达多样性带来的高歧义性及高昂计算成本。本文研究sLM在算术推理任务中表现不佳的原因，假设自然语言格式的变异性导致高歧义性。基于此，我们采用仅含公式的推理格式，将原本以自然语言表达的算术推理统一为数学方程。实验结果表明，该格式显著提升了sLM的算术推理能力，尤其在极小模型如T5-Tiny上效果明显。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) reasoning has enabled Large Language Model (LLM) to achieve remarkable performance in various NLP tasks, including arithmetic problem-solving. However, this success does not generalize to small language model (sLM) like T5, due to their limited capacity and absence of emergent abilities associated with larger models. Recent works to enhance sLM through knowledge distillation have yielded some improvements but still face significant limitations, particularly high ambiguity from the variability in natural language expressions and substantial computational costs. In this paper, we investigate why sLM perform poorly on arithmetic reasoning tasks and hypothesize that natural language format variability introduces high ambiguity for these smaller models. Based on this hypothesis, we conduct experiments with equation-only format, which is a reasoning format that unifies arithmetic reasoning previously expressed in natural language formats into mathematical equations. Experiment results demonstrate that equation-only format effectively boosts the arithmetic reasoning abilities of sLM, especially in very small models like T5-Tiny.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。