重新评估发现，TroVE在MATH上的优势主要来自更高算力，而非工具复用机制。
A Compute-Matched Re-Evaluation of TroVE on MATH
- 对比相同算力下，TroVE的工具复用效果不显著
- 调整选择机制后，准确率提升3%，但算力对齐后仅剩1%优势
- 适合关注算力公平性与模型机制可解释性的研究者
重用已有定理和公式是数学问题求解的核心，构成解决复杂挑战的关键基础。近期工作TroVE提出，代码生成的大语言模型可通过构建并复用高级工具箱，在MATH基准上取得优于仅直接生成代码的PRIMITIVE基线的表现。然而，近期分析（Berlot-Attwell等，2024）质疑其收益，指出所生成工具常为简单或极少被复用，可能源于自一致性或自纠错。本文重新评估TroVE在MATH上的表现，分析各模式影响，发现其优势并非来自这些机制，而是因使用了更高的计算预算。我们对原始实现中选择机制进行微调，使TroVE准确率提升3%。在算力匹配后，其性能优势仅剩1%，表明该工具箱方法在MATH上并无显著收益。
原文摘要 · Abstract (English)
Reusing established theorems and formulas is central to mathematical problem solving, serving as essential building blocks for tackling increasingly complex challenges. Recent work, TroVE, argues that code-generating Large Language Models (LLMs) can benefit similarly on the MATH benchmark by inducing and reusing higher-level toolboxes. By allocating computational budget across an ensemble of three modes -- directly generating code, creating tools, and reusing tools -- TroVE claims to outperform a PRIMITIVE baseline that only performs direct generation. However, recent analysis (Berlot-Attwell et al., 2024) casts doubt on these gains, noting that the tools created are often trivial or rarely reused, suggesting that improvements may stem from self-consistency or self-correction. In this work, we re-evaluate TroVE on MATH, analyze the impact of each of its modes, and show that its benefit does not come from these mechanisms, but simply from a higher computational budget spent for TroVE compared to PRIMITIVE. To this end, we also perform a small correction in the original implementation of TroVE's selection mechanism, boosting TroVE's performance on MATH by 3\% in accuracy. After matching for compute, the benefit of TroVE reduces to a marginal improvement of 1\%, suggesting that this toolbox approach does not provide a significant benefit on MATH.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。