用自回归方法解决资源分配中的约束难题，避免违规且性能更优。
Autoregressive Policy Optimization for Constrained Allocation Tasks
- 通过逐步采样方式生成满足约束的分配方案
- 在3个任务中均优于现有约束强化学习方法
- 适合需要严格遵守规则的金融与计算资源分配场景
资源分配任务要求在每个时间步将有限资源分配给一组实体，典型例子包括投资组合优化或计算负载在服务器间的分配。这类任务通常受线性约束限制，必须在所有时刻严格满足，例如投资者可能被要求任何投资周期内对某一行业投入不超过30%的资金。此类约束以复杂方式限制允许的分配动作空间，使学习能避免违规的策略变得困难。本文提出一种基于自回归过程的新方法，按顺序为每个实体采样分配方案，并引入新颖的去偏机制以抵消序列采样带来的初始偏差。我们在三个不同的约束分配任务上验证了该方法的优越性：投资组合优化、计算负载分配和一个合成基准任务。实验结果表明，该方法显著优于多种约束强化学习（CRL）方法。代码已开源：https://github.com/niklasdbs/paspo。
原文摘要 · Abstract (English)
Allocation tasks represent a class of problems where a limited amount of resources must be allocated to a set of entities at each time step. Prominent examples of this task include portfolio optimization or distributing computational workloads across servers. Allocation tasks are typically bound by linear constraints describing practical requirements that have to be strictly fulfilled at all times. In portfolio optimization, for example, investors may be obligated to allocate less than 30\% of the funds into a certain industrial sector in any investment period. Such constraints restrict the action space of allowed allocations in intricate ways, which makes learning a policy that avoids constraint violations difficult. In this paper, we propose a new method for constrained allocation tasks based on an autoregressive process to sequentially sample allocations for each entity. In addition, we introduce a novel de-biasing mechanism to counter the initial bias caused by sequential sampling. We demonstrate the superior performance of our approach compared to a variety of Constrained Reinforcement Learning (CRL) methods on three distinct constrained allocation tasks: portfolio optimization, computational workload distribution, and a synthetic allocation benchmark. Our code is available at: https://github.com/niklasdbs/paspo
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。