研究发现，数学推理中的大模型工具库其实很少复用，性能提升主要靠自我修正。
Library Learning Doesn't: The Curious Case of the Single-Use "Library"
- 通过分析两个工具库系统，发现函数复用率极低。
- 在miniF2F和MATH数据集上，复用率不足5%。
- 性能提升主要源于自我纠错和一致性机制，而非真正的工具复用。
大型语言模型（LLMs）的发展推动了数学推理中库学习系统的兴起。这些系统旨在学习可复用的工具库，如形式化的Isabelle引理或针对特定任务族的Python程序。尽管许多系统受到人类知识结构化启发，但当前方法是否真能学到可复用的工具库仍存疑。我们研究了两个数学库学习系统：LEGO-Prover和TroVE，二者均报告准确率提升。然而，在miniF2F和MATH数据集上，函数复用率极低，且消融实验表明，性能提升主要来自自校正与自一致性机制，而非工具复用。代码与数据已开源。
原文摘要 · Abstract (English)
Advances in Large Language Models (LLMs) have spurred a wave of LLM library learning systems for mathematical reasoning. These systems aim to learn a reusable library of tools, such as formal Isabelle lemmas or Python programs that are tailored to a family of tasks. Many of these systems are inspired by the human structuring of knowledge into reusable and extendable concepts, but do current methods actually learn reusable libraries of tools? We study two library learning systems for mathematics which both reported increased accuracy: LEGO-Prover and TroVE. We find that function reuse is extremely infrequent on miniF2F and MATH. Our followup ablation experiments suggest that, rather than reuse, self-correction and self-consistency are the primary drivers of the observed performance gains. Our code and data are available at https://github.com/ikb-a/curious-case
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。