解决推理时扩展中粒子滤波过早收敛问题,提升数学推理准确性。
Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scaling
- 引入熵退火与前瞻调制,动态保持推理路径多样性。
- 在多个数学基准上实现最高50%的任务奖励提升。
- 适合需要高可靠性推理的复杂任务场景。
推理时扩展(ITS)通过增加生成阶段计算量来提升语言模型性能。粒子滤波(PF)在复杂数学推理任务中表现优异,但受过程奖励模型引导时,常因早期给出过高置信分数而引发过早利用:模型盲目选择局部最优路径,剔除潜在正确假设,在有限计算预算下更易陷入次优解。这种现象称为粒子贫化,严重损害性能。本文分析根源在于粒子集缺乏多样性及对路径潜力评估能力不足。提出熵粒子滤波(ePF),融合两项新机制:熵退火(EA)通过监控搜索熵,在多样性下降时动态调整重采样分布以维持探索;前瞻调制(LaM)则引入预测性指导,基于后续状态评估当前状态潜力。在多个挑战性数学基准上,ePF显著优于强基线,任务奖励最高提升50%。该方法有效平衡了探索与利用,增强粒子滤波对复杂推理任务的鲁棒性。
原文摘要 · Abstract (English)
Inference-Time Scaling (ITS) improves language models by allocating more computation at generation time. Particle Filtering (PF) has emerged as a strong ITS method for complex mathematical reasoning tasks, but it is vulnerable when guided by process reward models, which often assign overconfident scores early in the reasoning process. This causes PF to suffer from premature exploitation: it myopically commits to locally promising trajectories, prunes potentially correct hypotheses, and converges to suboptimal solutions. This failure mode, known as particle impoverishment, is especially severe under constrained computational budgets. To address this, we analyze the problem and identify two root causes: a lack of diversity in the particle set due to overconfident resampling and consequent inability to assess the potential of a reasoning path. We introduce Entropic Particle Filtering (ePF), an algorithm that integrates two new techniques to solve these issues. The first technique, Entropic Annealing (EA), directly mitigates particle impoverishment by monitoring search diversity via entropy; when diversity drops, it intervenes by dynamically annealing the resampling distribution to preserve exploration. The second, an enhancement called Look-ahead Modulation (LaM), adds a predictive guide to evaluate a state's potential based on its successors. On several challenging math benchmarks, ePF significantly outperforms strong baselines and achieves up to a 50% relative improvement in task reward. Together, these methods improve PF's resilience by balancing the exploration of diverse solution spaces with the exploitation of high-reward regions, ultimately leading to higher-quality solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。