arXiv:2604.22504cs.IR2026-04

用束搜索负样本优化推荐目标,更贴合实际推荐效果

Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders

论文配图:Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders
图 1 · 摘自论文原文
  • 用束搜索负样本替代随机负样本,改变优化目标
  • 新方法在4个数据集上均达最优,提升Top-K推荐效果
  • 适合关注推荐系统性能优化的研究者和工程师

强化学习(RL)通过对比正负样本有效优化基于大语言模型(LLM)的推荐系统。实证表明,使用束搜索生成的负样本始终优于随机负样本,但其机制尚不明确。本文分析发现:(i)在二值奖励反馈下,使用组相对策略优化(GRPO)等价于最大化受试者工作特征曲线下面积(AUC),但该指标常与Top-$K$推荐目标不一致;(ii)将随机负样本替换为束搜索负样本可使优化目标向部分AUC转变,更贴近Top-$K$指标。为此,我们提出窗式部分AUC(WPAUC),将假阳性率(FPR)约束在区间[α, α+d]内,以更直接对齐Top-$K$性能。进一步提出高效的阈值调整窗式重加权(TAWin)RL方法,实现对目标Top-$K$表现的显式控制。四个真实数据集上的实验验证了理论并取得持续领先性能。

原文摘要 · Abstract (English)

Reinforcement learning (RL) effectively optimizes Large Language Model (LLM)-based recommenders by contrasting positive and negative items. Empirically, training with beam-search negatives consistently outperforms random negatives, yet the mechanism is not well understood. We address this gap by analyzing the induced optimization objective and show that: (i) Under binary reward feedback, optimizing LLM recommenders with Group Relative Policy Optimization (GRPO) is theoretically equivalent to maximizing the Area Under the ROC Curve (AUC), which is often misaligned with Top-$K$ recommendation; and (ii) Replacing random negatives with beam-search negatives reshapes the objective toward partial AUC, improving alignment with Top-$K$ metrics. Motivated by this perspective, we introduce Windowed Partial AUC (WPAUC), which constrains the false positive rate (FPR) to a window [$α,α+d$] to more directly align with Top-$K$ metrics. We further propose an efficient Threshold-Adjusted Windowed reweighting (TAWin) RL method for its optimization, enabling explicit control over the targeted Top-$K$ performance. Experiments on four real-world datasets validate the theory and deliver consistent state-of-the-art performance.

强化学习推荐系统大模型应用优化目标

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。