arXiv:2606.01081cs.LG2026-06

让智能决策在信息不全时仍能高效优化,适合复杂动态场景。

Decision-Focused On-Policy Learning for Contextual Linear Optimization with Partial Feedback

论文配图:Decision-Focused On-Policy Learning for Contextual Linear Optimization with Partial Feedback
图 1 · 摘自论文原文
  • 设计了基于混合梯度的在线学习策略,结合预测与决策优化。
  • 在多种任务中实现更低累积后悔,优于传统上下文带模型。
  • 适用于能源调度等真实场景,支持复杂概率建模。

决策聚焦学习(DFL)通过优化下游决策质量而非单纯预测精度来训练模型。针对上下文线性优化问题,现有方法多依赖离线数据和完整的成本向量观测。本文提出一种面向部分反馈的在线策略学习方法,推广标准的老虎机反馈设定。该方法学习一个随机‘预测-再优化’策略:从条件分布中采样成本向量,并求解相应的线性优化问题。为更新此分布模型,引入两组件混合梯度估计器:第一部分为得分函数估计器,提供无偏但可能高方差的策略梯度;第二部分为决策聚焦插值组件,利用辅助的潜在成本向量估计,借助下游优化结构提升信息量,估计越准越有效。理论证明平均平方策略梯度范数以 $ mathcal{O}(T^{-1/2})$ 收敛,达到标准非凸随机梯度下降速率。在 top-$k$ 选择、最短路径、组合定价及真实能源调度基准上实验表明,该混合梯度方法在使用高斯和更复杂的条件生成模型时,均显著降低累积后悔,优于上下文带基线。代码已公开于 https://github.com/Joeyetinghan/on-policy-bandit-dfl。

原文摘要 · Abstract (English)

Decision-focused learning (DFL) trains predictive models by optimizing downstream decision quality rather than standalone prediction accuracy. For contextual linear optimization, most existing DFL methods assume offline data and full observations of the objective cost vector. We develop an on-policy learning method for sequential contextual linear optimization under partial feedback, generalizing the standard bandit feedback setting. Our method learns a stochastic predict-then-optimize policy that samples a cost-vector prediction from a conditional distribution and solves the resulting downstream linear optimization problem. To update this distributional model, we introduce a two-component hybrid gradient estimator. The first component is a score function estimator, which provides an unbiased but potentially high-variance policy gradient estimate. The second is a decision-focused plug-in component that uses an auxiliary nuisance estimate of the latent cost vector to exploit the downstream optimization structure, becoming more informative as the estimate improves. We prove an $\mathcal{O}(T^{-1/2})$ bound on the average squared policy-gradient norm, matching the standard non-convex SGD rate. Experiments on top-$k$ selection, shortest path, combinatorial pricing, and a real-data energy-scheduling benchmark show that the hybrid gradient approach achieves lower cumulative regret than contextual-bandit-style baselines across all benchmarks, using both Gaussian and richer conditional generative models. Code is available at https://github.com/Joeyetinghan/on-policy-bandit-dfl.

决策优化在线学习部分反馈生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。