用动态价值间隔自训练，让大模型数学推理更准且无需依赖强模型或人工标注。
SPPD: Self-training with Process Preference Learning Using Dynamic Value Margin
- 基于过程的马尔可夫决策过程，每一步计算动态价值间隔优化偏好。
- 70亿参数模型在数学基准上表现超越现有方法，跨域任务也更稳健。
- 无需模型蒸馏或人工标注，适合资源有限但想提升推理能力的研究者。
近期提升大语言模型（LLM）的数值与逻辑推理能力成为研究热点。现有方法存在诸多局限：推理阶段技术（如思维链）依赖提示选择和预训练知识；句子级监督微调（SFT）与直接偏好优化（DPO）难以保证步骤级数学正确性，且依赖更强模型蒸馏或人工标注；强化学习（RL）方法则成本高、训练不稳定。为此，我们提出自训练框架 SPPD（Self-training with Process Preference Learning using Dynamic Value Margin）。SPPD 基于过程型马尔可夫决策过程（MDP）与贝尔曼最优方程，推导出步骤级偏好优化的动态价值间隔，通过树状自采样生成模型响应，完全不依赖其他模型的蒸馏。此外，我们理论证明 SPPD 在奖励约束下等价于策略梯度方法。70亿参数模型的实验表明，SPPD 在域内与域外数学基准上均表现优异。代码已开源。
原文摘要 · Abstract (English)
Recently, enhancing the numerical and logical reasoning capability of Large Language Models (LLMs) has emerged as a research hotspot. Existing methods face several limitations: inference-phase techniques (e.g., Chain of Thoughts) rely on prompt selection and the pretrained knowledge; sentence-level Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) struggle with step-wise mathematical correctness and depend on stronger models distillation or human annotations; while Reinforcement Learning (RL) approaches incur high GPU memory costs and unstable training. To address these, we propose \textbf{S}elf-training framework integrating \textbf{P}rocess \textbf{P}reference learning using \textbf{D}ynamic value margin (SPPD). SPPD leverages a process-based Markov Decision Process (MDP) and Bellman optimality equation to derive \textbf{dynamic value margin} on step-level preference optimization, which employs tree-based self-sampling on model responses \textbf{without any distillation} from other models. Furthermore, we theoretically prove that SPPD is \textbf{equivalent to on-policy policy gradient methods} under reward constraints. Experiments on 7B-scale models demonstrate superior performance across in-domain and out-domain mathematical benchmarks. We open-source our code at \href{https://anonymous.4open.science/r/SSDPO-D-DCDD}{https://anonymous.4open.science/r/SPPD-DCDD}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。