用可微模拟器加速复杂决策优化,一次反向传播搞定所有敏感性分析。
SNAPO: Smooth Neural Adjoint Policy Optimization for Optimal Control via Differentiable Simulation
- 将神经策略嵌入可微模拟器,用平滑近似替代硬约束
- 单次反向传播完成策略训练与数千项敏感性计算
- 适合需要快速敏感性分析的能源、金融、制药等高维决策场景
许多现实问题需在不确定性下进行序列决策:何时注入或提取天然气、每月如何调整养老金投资组合、制药反应链应采用何种温度曲线。动态规划能精确求解小规模问题,但状态维度增加时计算量指数级增长。黑箱强化学习可处理高维状态,但训练慢且无法提供敏感性信息。本文提出SNAPO(平滑神经伴随策略优化),将神经策略嵌入已知的可微模拟器中,用平滑近似替代硬约束,并通过一次伴随反向传播精确计算目标函数对所有策略参数和输入的梯度。我们在三个领域验证SNAPO:天然气储气(训练时间不足1分钟,每条远期曲线敏感性无需额外成本)、养老金资产负债管理(敏感性计算速度比传统扰动法快6.5倍至200倍,随风险因子数量线性扩展)、制药制造(4单元流程链跨单元敏感性分析,5次伴随反向传播内完成20项ICH Q8监管敏感性,耗时74.5毫秒)。所有敏感性均由训练所用的同一反向传播生成,计算成本仅相当于一次反向传播,与敏感性数量无关。
原文摘要 · Abstract (English)
Many real-world problems require sequential decisions under uncertainty: when to inject or withdraw gas from storage, how to rebalance a pension portfolio each month, what temperature profile to run through a pharmaceutical reactor chain. Dynamic programming solves small instances exactly but scales exponentially in state dimensions. Black-box reinforcement learning handles high-dimensional states but trains slowly and produces no sensitivities. We introduce SNAPO (Smooth Neural Adjoint Policy Optimization), a framework that embeds a neural policy inside a known, differentiable simulator, replaces hard constraints with smooth approximations, and computes exact gradients of the objective with respect to all policy parameters and all inputs in a single adjoint pass. We demonstrate SNAPO on three domains: natural gas storage (training in under a minute, 365 forward curve sensitivities at no additional cost per sensitivity), pension fund asset-liability management (6.5x-200x sensitivity speedup over bump-and-revalue, scaling with the number of risk factors), and pharmaceutical manufacturing (cross-unit sensitivities through a 4-unit process chain, with 20 ICH Q8 regulatory sensitivities from 5 adjoint passes in 74.5 milliseconds). All sensitivities are produced by the same backward pass that trains the policy, at a cost proportional to one reverse pass regardless of how many sensitivities are computed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。