用搜索与强化学习结合,让模型在解题时自动优化建模步骤。
StarOR: Synergizing Tree Search and Test-Time Reinforcement Learning for Optimization Modeling

- 通过MCTS和GRPO动态调整建模策略,每步决策都可优化
- 在5个基准上超越现有方法,40亿参数模型已达顶尖水平
- 无需标注数据,适合复杂优化问题的实时建模
优化建模具有固有的层次结构,需要精确的符号决策序列。传统基于学习的自动化建模方法依赖大规模标注数据训练策略,但难以适应新问题分布;而一次性生成在层次化建模中易因早期错误导致无效公式。测试时扩展提供新路径,通过额外实例级计算实现结构探索;但现有搜索方法通常采用固定策略,重复采样继承相同偏差,且对中间决策的信用分配有限。为此,我们提出StarOR,一种将蒙特卡洛树搜索(MCTS)与测试时强化学习相结合的协同框架。StarOR将建模过程分解为四个阶段,在每个非终止节点通过GRPO更新临时LoRA适配器。利用MCTS生成的兄弟节点作为局部比较集,将搜索期探索转化为实例特定策略精炼。此外,无监督多维度奖励系统为中间建模决策提供细粒度反馈,无需真实标签。在五个优化基准上的实验表明,即使使用40亿参数主干模型,StarOR仍达到当前最优性能,优于现有方法及前沿大模型。
原文摘要 · Abstract (English)
Optimization modeling is inherently hierarchical, requiring a precise sequence of symbolic commitments. Traditional learning-based automated optimization modeling methods improve modeling policies through large-scale annotated or curated training data, but are costly to adapt to new problem distributions. Meanwhile, one-shot generation remains brittle in hierarchical modeling, where early symbolic errors can propagate into invalid formulations. Test-time scaling offers a promising alternative by enabling structural exploration with additional instance-level computation; however, existing search-based methods typically rely on a fixed policy, causing repeated rollouts to inherit similar modeling biases and providing limited credit assignment for intermediate decisions. To address these limitations, we propose StarOR, a synergistic search-and-adaptation framework that couples MCTS with Test-Time Reinforcement Learning for optimization modeling. StarOR decomposes the modeling process into four stages and updates a transient LoRA adapter via GRPO at each non-terminal node. By using MCTS-generated siblings as local comparison sets, StarOR transforms search-time exploration into instance-specific policy refinement. Moreover, an unsupervised multi-faceted reward system provides fine-grained feedback for intermediate formulation decisions without ground-truth labels. Experiments across five optimization benchmarks show that StarOR achieves state-of-the-art performance even with a 4B backbone, outperforming existing methods and the frontier LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。