用双时间尺度强化学习提升运营系统抗冲击能力
Two-Timescale Hierarchical Reinforcement Learning for Resilient Operations
- 分长短期策略同步更新，协同优化决策
- 长期与短期联合适应，利润提升9.2%~11.8%
- 适合已有分级决策结构的实体企业使用
全球运营中意外冲击频发，需动态调整决策规则以应对市场与运行条件变化。许多系统具有层级结构，长期与短期决策共同追求同一目标。本文研究如何通过层次强化学习增强韧性，联合适应这些相互依赖的规则。提出一种双时间尺度层次强化学习框架，分别在不同时间尺度上更新长期与短期策略。由于策略相互依赖，采用同步更新机制，并首次证明了耦合双时间尺度学习的收敛性。在T个周期内，策略平均偏离最优策略对的差距为O(T⁻¹/²)，当错误决策导致更明显利润损失时可优化至O(log T/T)。在二手车案例中，库存补货为长期决策，客户到店定价为短期决策。相较于最强部分自适应基准，该框架在联合供需冲击下平均利润提升9.2%，在持续冲击场景下提升11.8%，且利润轨迹更稳定。短期适应可应对常规季节性及单边扰动，但面对联合冲击仍不足；长期适应则为短期决策创造有利条件。联合适应在冲击与恢复期均实现更高更稳的收益。因多数组织已有层级规划，该框架无需改变现有决策结构即可增强韧性。
原文摘要 · Abstract (English)
Unexpected shocks recur in global operations, requiring decision rules that adapt as market and operating conditions change. Many operational systems also have hierarchical structures in which long-term and short-term decisions pursue a shared objective. We study how hierarchical reinforcement learning can strengthen resilience by adapting these interdependent rules jointly. We develop a two-timescale hierarchical reinforcement learning framework that adapts long-term and short-term policies at their respective time scales. Because the policies are interdependent, we synchronize their updates and prove, to our knowledge, the first convergence guarantees for coupled two-timescale learning. Over $T$ periods, our policies' average gap from an optimal policy pair is $O(T^{-1/2})$, improving to $O(\log T/T)$ when poor decisions produce clearer profit losses. In a used-car case study, inventory replenishment is the long-term decision and customer-arrival pricing the short-term decision. Relative to the strongest partially adaptive benchmark, the framework increases mean profit by $9.2\%$ under joint demand-supply shocks and by $11.8\%$ under a prolonged shock scenario, while maintaining a more stable profit trajectory over time. Short-term adaptation addresses routine seasonality and one-sided disruptions by responding immediately to changing conditions. Under joint demand-supply shocks, however, it is insufficient alone; long-term adaptation is also needed to create favorable conditions for short-term decisions. Joint adaptation thus yields higher and more stable profits through disruption and recovery. Because many organizations already use hierarchical planning, the framework strengthens operational resilience without altering existing decision structures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。