用延迟反馈优化外卖调度的权重，提升效率不降体验。
Multi-Agent Reinforcement Learning from Delayed Marketplace Feedback for Objective-Weight Adaptation in Three-Sided Dispatch
- 用离线学习的策略调整调度权重，不改动原有优化器。
- 实验显示批量处理增加，骑手成本降低，客户体验不变。
- 适合做物流调度优化和在线决策系统的从业者参考。
三边外卖平台的调度问题天然适合基于世界反馈的强化学习：决策效果由延迟的运营结果（如配送速度、骑手利用率、商家拥堵）评估。我们在DoorDash部署了一个强化学习系统，通过延迟信号动态调整调度目标权重。该系统不替换原有的组合分配优化器，而是通过从历史数据中学习的门店级策略，选择一个离散乘数来调节优化器在配送质量与打包效率之间的权衡。这种接口使系统能在噪声大、延迟高且耦合的反馈下进行离线策略学习，同时保留生产可行性约束和运营安全机制。我们使用集中式离线数据训练共享价值函数，采用分布式门店执行，结合双Q-learning目标和保守正则化，以减少分布外值估计偏差。在生产环境的切换实验中，离线训练的策略提升了批量处理率，降低了骑手侧时间成本，且未影响客户感知的配送质量。结果表明，真实经济与物流系统的世界反馈可用于安全地在线调整决策策略。
原文摘要 · Abstract (English)
Dispatch in three-sided marketplaces provides a natural setting for reinforcement learning from world feedback: decisions are evaluated by delayed operational outcomes such as delivery speed, courier utilization, and merchant congestion. We present a deployed reinforcement learning system at DoorDash that adapts dispatch objective weights in a large-scale food-delivery marketplace using delayed signals. Rather than replacing the combinatorial assignment optimizer, a store-level policy learned from logged marketplace data selects a discrete multiplier that shifts the dispatch optimizer's tradeoff between delivery quality and batching efficiency. This interface enables offline policy learning under noisy, delayed, and coupled feedback while preserving production feasibility constraints and operational safeguards. We train a shared value function using centralized offline data and decentralized store-level execution, with Double Q-learning targets and a conservative regularizer to reduce out-of-distribution value overestimation. In a production switchback experiment, the offline-trained policy increases batching and reduces courier-side time costs without degrading customer-facing delivery quality. Results illustrate how world feedback from a live economic and logistics system can be used to safely adapt decision policies online.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。