用数学方法解析大模型解题思路,看清每步推理的贡献度。
SalaMAnder: Shapley-based Mathematical Expression Attribution and Metric for Chain-of-Thought Reasoning
- 基于谢林值分析每个数学表达式在推理中的贡献
- 提出新指标CoSP,与模型性能强相关且可解释
- 适合研究大模型推理机制或优化提示词的人
链式思维(CoT)提示显著提升了大语言模型(LLMs)的数学推理能力,但其内在机制仍不明确。本文提出SalaMAnder(基于谢林值的数学表达式归因与评估指标),一种理论严谨的方法和数学严格评估指标,用于量化少样本CoT推理中各组件的贡献。我们采用谢林值进行数学表达式归因,并设计高效分层采样算法,大幅降低计算复杂度。此外,通过协方差分析构建了CoSP(谢林正值基数)指标。在多个主流LLM模型和数学基准上的综合验证表明,SalaMAnder框架内的CoSP指标与模型性能呈现稳健单调相关性,不仅为现有少样本CoT的实证成功提供理论解释,还建立了提示构造优化的数学严谨原则。进一步验证了解释的可靠性,并统一了以往研究的洞见。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) prompting enhances the math reasoning capability of large language models (LLMs) to a large margin. However, the mechanism underlying such improvements remains unexplored. In this paper, we present \textbf{SalaMAnder} (\textbf{S}h\textbf{a}p\textbf{l}ey-b\textbf{a}sed \textbf{M}athematical Expression \textbf{A}ttribution a\textbf{nd} M\textbf{e}t\textbf{r}ic), a theoretically grounded methodology as well as a mathematically rigorous evaluation metric for quantifying component-level contributions in few-shot CoT reasoning. Concretely, we leverage the Shapley value for mathematical expression attribution and develop an efficient stratified sampling algorithm that significantly reduces the computational complexity. Besides, we develop the \textbf{CoSP} (\textbf{C}ardinality \textbf{o}f \textbf{S}hapley \textbf{P}ositives) metric through covariance analysis. Comprehensive validation across popular LLM models and diverse mathematical benchmarks demonstrates that the CoSP metric within our SalaMAnder framework exhibits a robust monotonic correlation with model performance, not only providing theoretical explanations for the empirical success of existing few-shot CoT but also establishing mathematically rigorous principles for prompt construction optimization. Furthermore, we verify the reliability of the explanation, based on which we unify the insights of previous work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。