用分层强化学习解决多年度基建预算管理难题
Hierarchical Deep Reinforcement Learning Framework for Multi-Year Asset Management Under Budget Constraints
- 分两层决策：先定年度预算，再选维护资产
- 在10-20个污水区上收敛更快，近似最优
- 适合大型基建规划、需严格控预算的场景
预算是基础设施资产管理中的关键环节，直接影响成本效益与可持续性。然而，组合动作空间、资产退化差异、严苛预算约束及环境不确定性，严重制约了现有方法的可扩展性。本文提出一种分层深度强化学习框架，专为多年度基础设施规划设计。该方法将问题分解为两个层级：高层预算规划器在显式可行性范围内分配年度预算；低层维护规划器在预算内优先排序资产。通过在分层软演员-评论家框架中集成线性规划投影，有效缓解动作空间指数级增长问题，并确保严格的预算合规性。案例研究涵盖10、15和20个污水区的管网系统，结果表明，相比传统深度Q学习和改进遗传算法，本方法收敛更迅速，可扩展性强，且随网络规模扩大仍能持续输出近优解。
原文摘要 · Abstract (English)
Budget planning and maintenance optimization are crucial for infrastructure asset management, ensuring cost-effectiveness and sustainability. However, the complexity arising from combinatorial action spaces, diverse asset deterioration, stringent budget constraints, and environmental uncertainty significantly limits existing methods' scalability. This paper proposes a Hierarchical Deep Reinforcement Learning methodology specifically tailored to multi-year infrastructure planning. Our approach decomposes the problem into two hierarchical levels: a high-level Budget Planner allocating annual budgets within explicit feasibility bounds, and a low-level Maintenance Planner prioritizing assets within the allocated budget. By structurally separating macro-budget decisions from asset-level prioritization and integrating linear programming projection within a hierarchical Soft Actor-Critic framework, the method efficiently addresses exponential growth in the action space and ensures rigorous budget compliance. A case study evaluating sewer networks of varying sizes (10, 15, and 20 sewersheds) illustrates the effectiveness of the proposed approach. Compared to conventional Deep Q-Learning and enhanced genetic algorithms, our methodology converges more rapidly, scales effectively, and consistently delivers near-optimal solutions even as network size grows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。