用人类解题策略提升大模型算术能力,发现其学习模式与人相似。
Explaining and Tuning Transformer-based LLMs in Arithmetic Tasks with Human Strategies

- 将算术任务拆解为子任务,分析各环节学习顺序
- 应用人类有效的解题策略后,模型准确率显著提升
- 揭示大模型与人类在算术学习中可能存在共同认知机制
基于Transformer的大语言模型在自然语言处理任务中表现优异,但在基础算术等看似简单的问题上表现不佳,引发对其可靠性、安全性及伦理部署的担忧。本研究发现,对未经优化的变压器模型进行整数算术训练时,可借鉴人类学习者有效的方法来提升性能。我们首先将算术任务分解为明确的子任务,并通过损失收敛顺序分析和消融实验评估各子任务影响。结果表明,大模型的学习模式与人类相似:简单子任务学习速度更快,复杂子任务更慢。此外,通过引入人类验证有效的解题策略与认知增强方法,成功提升了大模型的准确性。该结果暗示基于Transformer的大模型在算术学习中可能与人类共享认知过程。最后,我们通过大量准确率提升实验、可视化验证与基于解释的分析,全面展示了该方法的有效性。本研究结合可解释AI(XAI)验证,探索了大模型与人类学习者的潜在相似性,有助于增强大模型在高风险关键场景中的可信度。
原文摘要 · Abstract (English)
Transformer-based large language models (LLMs) continue to achieve state-of-the-art performance across various natural language processing tasks. However, their subpar performance on seemingly elementary problems, such as basic arithmetic, raises concerns about model reliability, safety, and ethical deployment. In this study, we demonstrate that the performance of a vanilla Transformer model trained on integer arithmetic tasks can be improved using methods effective for human learners. We begin by decomposing the arithmetic task into well-defined subtasks and conducting loss convergence order analysis together with ablation studies for each subtask. Our findings reveal that LLMs exhibit learning patterns similar to those of human learners, with a faster learning speed for simpler subtasks compared to more complex ones. In addition, we successfully improved the accuracy of LLMs by applying problem-solving strategies and cognitive empowerment methods shown to enhance the performance of human learners. This suggests that transformer-based LLMs may share cognitive processes with human learners in arithmetic. Lastly, we provide a comprehensive demonstration of our method's effectiveness, including significant accuracy improvement experiments, visualization verification, and explanation-based analysis to illuminate the intricacies of LLMs in arithmetic learning. In general, this work explores the potential similarities between transformer-based LLMs and human learners, supported by explainable AI (XAI) verifications, ultimately fostering trust in LLMs for critical and high-stakes applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。