让智能体在线生成新动作，平衡成本与收益。
Online Decision Making with Generative Action Sets
- 用上下界策略同时决定选动作和何时生成新动作。
- 在医疗问答数据集上表现优于基线方法，生成质量更优。
- 首个证明可扩展动作空间下亚线性遗憾的算法，适合动态决策场景。
随着生成式AI的发展，决策代理可在在线学习过程中动态创建新动作，但动作生成通常需支付一次性成本，必须权衡潜在收益。本文研究一种在线学习问题：代理可在任意时间步通过支付固定成本生成新动作，这些动作将永久可用于后续决策。核心挑战在于学习最优的双重决策序列——选择动作时机与生成新动作的时机，且三者间存在三角权衡：利用、探索与创造。为此，我们提出一种双重乐观算法，使用置信下界（LCB）进行动作选择，使用置信上界（UCB）进行动作生成。在医疗问答数据集上的实证评估表明，该方法在生成质量与策略性能之间取得了更优平衡。理论分析显示，该算法实现最优遗憾界 $O(T^{ rac{d}{d+2}}d^{ rac{d}{d+2}} + d oot{T} oot{ ext{log } T})$，首次为可扩展动作空间的在线学习提供了亚线性遗憾保证。
原文摘要 · Abstract (English)
With advances in generative AI, decision-making agents can now dynamically create new actions during online learning, but action generation typically incurs costs that must be balanced against potential benefits. We study an online learning problem where an agent can generate new actions at any time step by paying a one-time cost, with these actions becoming permanently available for future use. The challenge lies in learning the optimal sequence of two-fold decisions: which action to take and when to generate new ones, further complicated by the triangular tradeoffs among exploitation, exploration and $\textit{creation}$. To solve this problem, we propose a doubly-optimistic algorithm that employs Lower Confidence Bounds (LCB) for action selection and Upper Confidence Bounds (UCB) for action generation. Empirical evaluation on healthcare question-answering datasets demonstrates that our approach achieves favorable generation-quality tradeoffs compared to baseline strategies. From theoretical perspectives, we prove that our algorithm achieves the optimal regret of $O(T^{\frac{d}{d+2}}d^{\frac{d}{d+2}} + d\sqrt{T\log T})$, providing the first sublinear regret bound for online learning with expanding action spaces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。