通过剔除推理轨迹中的低效片段,提升大模型数学推理准确率。
SLIM: Subtrajectory-Level Elimination for More Effective Reasoning
- 将推理过程拆分为子轨迹,用五项人工标准识别低效部分。
- 剔除25.9%的低效子轨迹后,仅用三分之二数据达58.92%准确率。
- 适合资源受限场景,尤其提升小样本下数学推理性能。
近期大型语言模型在复杂推理方面取得显著进展,尤其得益于测试时扩展(test-time scaling)。典型如o1/o3/o4系列和DeepSeek-R1,在响应查询时生成长推理轨迹,包含探索、反思、回溯与自验证。然而,基于此类轨迹进行微调未必最优。本研究发现,推理轨迹中并非所有组件均有益,部分甚至可能损害整体表现。为此,我们将推理轨迹划分为子轨迹,提出“5+2”框架:(1) 基于五项人类设定标准系统识别低效子轨迹;(2) 评估其与后续内容的独立性,确保剔除不影响推理连贯性。进一步设计采样算法,筛选出低效子轨迹最少的数据。实验表明,该方法使推理中低效子轨迹减少25.9%。在仅使用三分之二训练数据条件下,微调Qwen2.5-Math-7B在高难度数学基准上平均准确率达58.92%,优于完整数据集的58.06%,并超越开源数据集表现。在资源受限环境下亦验证有效,多种推理词元限制下性能均有提升。
原文摘要 · Abstract (English)
In recent months, substantial progress has been made in complex reasoning of Large Language Models, particularly through the application of test-time scaling. Notable examples include o1/o3/o4 series and DeepSeek-R1. When responding to a query, these models generate an extended reasoning trajectory, during which the model explores, reflects, backtracks, and self-verifies before arriving at a conclusion. However, fine-tuning models with such reasoning trajectories may not always be optimal. Our findings indicate that not all components within these reasoning trajectories contribute positively to the reasoning process; in fact, some components may affect the overall performance negatively. In this study, we divide a reasoning trajectory into individual subtrajectories and develop a "5+2" framework to: (1) systematically identify suboptimal subtrajectories within the reasoning trajectory based on five human-established criteria; (2) assess the independence of the suboptimal subtrajectories identified in (1) from the subsequent content, ensuring that their elimination does not compromise overall flow and coherence of the reasoning process. Additionally, a sampling algorithm, built upon the "5+2" framework, is employed to select data whose reasoning process is free from suboptimal subtrajectories to the highest degree. Experimental results demonstrate that our method can reduce the number of suboptimal subtrajectories by 25.9\% during the inference. Furthermore, our method achieves an average accuracy of 58.92\% on highly challenging math benchmarks with only two thirds of training data, surpassing the average accuracy of 58.06\% achieved with the entire data, and outperforming open-source datasets, when fine-tuning Qwen2.5-Math-7B. Finally, We validated our method under resource constraints and observed improved performance across various inference token limits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。