通过分析推理片段的贡献度,筛选重要推理过程提升模型效率与准确率。
Segment-Level Attribution for Selective Learning of Long Reasoning Traces
- 用归因分析量化每个词元对答案的影响,生成段落级贡献度指标。
- 在多个模型和数据集上,准确率提升且输出更简洁,减少冗余内容。
- 适合想优化长链推理模型训练效率的研究者或开发者使用。
大型推理模型(LRMs)通过生成长链思维(CoTs)实现强大推理能力,但其中仅小部分内容真正影响答案预测,多数为重复或不完整的表达。这种冗余在监督微调(SFT)后被进一步放大,导致模型模仿低效的冗长模式,降低性能。为此,本文引入集成梯度归因,量化每个词元对最终答案的影响,并聚合为两个段落级指标:(1) 归因强度,衡量整体影响幅度;(2) 方向一致性,判断段内归因是否统一(高一致性)或混合(中一致性)。基于此,提出一种段落级选择性学习框架,识别具有高归因强度但中等一致性的关键推理段落,这些段落体现反思性而非浅层推理。框架对这些重要段落进行选择性SFT,同时屏蔽无关段落的损失。多模型、多数据集实验表明,该方法提升了准确率与输出效率,使长推理链的利用更有效(代码与数据见https://github.com/SiyuanWangw/SegmentSelectiveSFT)。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) achieve strong reasoning performance by generating long chains of thought (CoTs), yet only a small fraction of these traces meaningfully contributes to answer prediction, while the majority contains repetitive or truncated content. Such output redundancy is further propagated after supervised finetuning (SFT), as models learn to imitate verbose but uninformative patterns, which can degrade performance. To this end, we incorporate integrated gradient attribution to quantify each token's influence on final answers and aggregate them into two segment-level metrics: (1) \textit{attribution strength} measures the overall attribution magnitude; and (2) \textit{direction consistency} captures whether tokens' attributions within a segment are uniformly positive or negative (high consistency), or a mixture of both (moderate consistency). Based on these two metrics, we propose a segment-level selective learning framework to identify important segments with high attribution strength but moderate consistency that indicate reflective rather than shallow reasoning. The framework then applies selective SFT on these important segments while masking loss for unimportant ones. Experiments across multiple models and datasets show that our approach improves accuracy and output efficiency, enabling more effective learning from long reasoning traces~\footnote{Code and data are available at https://github.com/SiyuanWangw/SegmentSelectiveSFT}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。