解决推荐系统中反馈延迟问题，让算法更准更稳。
Linear and Neural Dueling Bandits with Delayed Feedback
- 在损失函数中嵌入逆概率加权，修正延迟反馈偏差。
- 线性模型理论证明后悔界为O(d*sqrt(T))，神经网络也达次线性。
- 适合做推荐、大模型对齐等需处理延迟反馈的场景。
上下文双人博弈老虎机是基于偏好的决策核心，在推荐系统和大语言模型对齐中有重要应用。然而，现有算法依赖即时反馈的理想假设，这在实际场景（如提示优化）中常不成立。此设置带来独特挑战：与线性老虎机不同，双人博弈估计器无闭式解，直接套用标准加权技术会导致偏差。为此，我们形式化了带随机延迟反馈的上下文双人博弈老虎机问题，提出两种新算法：线性（LDB-DF）与神经（NDB-DF）双人博弈老虎机。核心是将逆概率加权（IPW）机制直接融入损失函数，实现延迟或缺失反馈的无偏修正。我们提供完整理论分析，证明线性情形下后悔界为O(d*sqrt(T))，神经情形也具次线性保证。在模拟与真实数据集上的大量实验验证了方法的有效性。
原文摘要 · Abstract (English)
Contextual dueling bandits form a cornerstone of preference-based decision-making, with critical applications in recommender systems and large language model alignment. However, standard algorithms rely on the idealized assumption of immediate feedback, a condition frequently violated in real-world scenarios such as prompt optimization. This setting introduces a unique theoretical challenge: unlike linear bandits, dueling bandit estimators lack closed-form solutions, rendering naive adaptations of standard weighting techniques biased. To address this, we formalize the problem of Contextual Dueling Bandits with Stochastic Delayed Feedback and propose two novel algorithms: Linear (LDB-DF) and Neural (NDB-DF) Dueling Bandits with Delayed Feedback. Central to our approach is a novel estimator that integrates an Inverse Probability Weighting (IPW) mechanism directly into the loss function, ensuring unbiased correction for delayed or missing feedback. We provide comprehensive theoretical analysis, establishing an O(d*sqrt(T)) regret bound for the linear setting and sub-linear guarantees for the neural setting. Extensive experiments on both simulated and real-world datasets demonstrate the effectiveness of our propose.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。