arXiv:2412.00569cs.LGcs.IR2024-12

在支付系统中用非均匀探索提升决策,避免性能下降。

Contextual Bandits in Payment Processing: Non-uniform Exploration and Supervised Learning

  • 结合监督学习与非均匀探索,提升即时决策效果。
  • 发现策略迭代中因奖励分布变化导致性能下降。
  • 适合关注长期稳定性的工业级强化学习应用者。

决策系统中的均匀随机探索虽支持离策略学习,但代价高昂,不适用于实际场景;而非均匀探索虽即时表现更好,却难以支持离策略学习。近期研究提出利用回归预言机(regression oracle)来弥合这一差距。本文在大型全球支付平台Adyen的实际工业场景下分析该方法,其特点包括批量日志延迟反馈、短期记忆和动态动作空间,基于经验风险最小化(ERM)框架。分析显示,尽管回归预言机显著提升性能,但因其算法假设僵化,引发新问题:随着策略优化,后续生成的策略反而表现更差,这是由于奖励分布偏移及训练数据类别不平衡加剧所致。这种退化现象发生在训练数据其他方面持续改善的情况下,导致策略迭代中性能下降。我们进一步发现回归预言机可能引发‘振荡效应’——其对概率估计的影响使后续策略模型的可实现性不稳定,造成性能在迭代间波动。研究强调需开发更灵活的算法,以保留回归预言机优势的同时避免长期性能不稳。

原文摘要 · Abstract (English)

Uniform random exploration in decision-making systems supports off-policy learning via supervision but incurs high regret, making it impractical for many applications. Conversely, non-uniform exploration offers better immediate performance but lacks support for off-policy learning. Recent research suggests that regression oracles can bridge this gap by combining non-uniform exploration with supervised learning. In this paper, we analyze these approaches within a real-world industrial context at Adyen, a large global payments processor characterized by batch logged delayed feedback, short-term memory, and dynamic action spaces under the Empirical Risk Minimization (ERM) framework. Our analysis reveals that while regression oracles significantly improve performance, they introduce challenges due to rigid algorithmic assumptions. Specifically, we observe that as a policy improves, subsequent generations may perform worse due to shifts in the reward distribution and increased class imbalance in the training data. This degradation occurs de spite improvements in other aspects of the training data, leading to decreased performance in successive policy iterations. We further explore the long-term impact of regression oracles, identifying a potential "oscillation effect." This effect arises when regression oracles influence probability estimates and the realizability of subsequent policy models, leading to fluctuations in performance across iterations. Our findings highlight the need for more adaptable algorithms that can leverage the benefits of regression oracles without introducing instability in policy performance over time.

强化学习支付系统策略优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。