用强化学习让大模型自动生成可复用的优化求解器,提升效率并降低推理成本。
Beyond Inference-Time Search: Reinforcement Learning Synthesizes Reusable Solvers

- 通过强化学习训练代码模型,使其学会生成符合约束的模拟退火求解模板。
- 在SDS问题上达到5.0%误差,比基线方法降低23.7个百分点,执行成本降91倍。
- 生成的求解器可冻结复用,适合需要高效求解的组合优化场景。
大语言模型通常将组合优化问题作为推理阶段的独立实例处理,通过采样、搜索或重复提示解决。本文探索是否可通过强化学习将部分推理开销转移至代码模型权重中,使模型为整个问题族合成可复用的求解器。以受控的协同依赖选择(SDS)为例,该问题暴露了贪心启发式因局部信号和严格可行性约束而看似有效但不可靠的缺陷。在相同框架下,Best-of-64基线采样仅能达到约28.7%的全局虚拟最优解差距;代码审计显示基模常调用模拟退火模板但错误实现梅特罗波利斯接受规则。使用组相对策略优化(GRPO)对Qwen2.5-Coder-14B-Instruct进行微调,采用可行性门控奖励与轻量结构支架,所获策略在99.8%的可行输出中收敛至约束感知的模拟退火模板,实现5.0%的差距,且后处理执行/搜索成本仅为累计Best-of-64评估的1/91。一次编译验证表明,每个种子冻结一个最佳求解器即可在测试集上保持高竞争力;跨域评估在作业车间调度任务上也显示出较窄但积极的迁移迹象。负向消融实验揭示该方法的局限性:标准稳定器会损害性能,软可行性门失效,结果仍对奖励归一化及领域特定设计敏感。
原文摘要 · Abstract (English)
Large language models (LLMs) typically approach combinatorial optimization as an inference-time procedure, solving each instance separately through sampling, search, or repeated prompting. We ask whether reinforcement learning can instead shift part of this reasoning cost into the weights of a code LLM, so that the model synthesizes a reusable solver for an entire problem family. We study this question on Synergistic Dependency Selection (SDS), a controlled variant of constrained Quadratic Knapsack designed to expose a specific failure mode: local signals and strict feasibility constraints make greedy heuristics attractive but unreliable. Under identical scaffolding, Best-of-64 base-model sampling saturates at an approximately 28.7% gap to the global Virtual Best Solver (VBS); code audits show that the base model often retrieves Simulated Annealing templates but misimplements the Metropolis acceptance rule. We fine-tune Qwen2.5-Coder-14B-Instruct with Group Relative Policy Optimization (GRPO) using a feasibility-gated reward and light structural scaffolding. The resulting policy converges to a constraint-aware Simulated Annealing template in 99.8% of feasible SDS outputs, achieves a 5.0% gap to that VBS, and is 91 times cheaper in post-generation execution/search cost than cumulative Best-of-64 evaluation. A compile-once check shows that one best frozen solver per seed remains highly competitive when reused unchanged across the SDS test set, while an additional-domain evaluation on Job Shop Scheduling provides narrower but positive evidence that the scaffold transfers beyond SDS. Negative ablations reveal the limits of this recipe: standard stabilizers degrade performance, a soft feasibility gate fails, and results remain sensitive to reward normalization and domain-specific design choices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。