用强化学习自动调优推荐系统的权重,提升用户参与度。
A Production-Ready RL Framework for Personalized Utility Tuning with Pareto Sweeping in Pinterest Recommender Systems

- 将权重调整建模为单步价值型强化学习问题,根据请求上下文选择最优权重向量。
- 在线实验显示,成功会话率提升0.13%,显著优于基线模型。
- 支持实时帕累托前沿扫描,便于决策者快速选择最优策略,适合生产环境使用。
大规模推荐系统通过融合多个预测结果生成单一效用得分来处理多目标权衡。尽管效用层可独立于排序器更新,但权重调优仍主要依赖人工、全局应用、适应慢且难以随业务需求变化而管理。本文提出PRL-PUTS:一个面向个性化效用权重调优的生产就绪、排序器无关的强化学习框架,采用帕累托扫掠策略。将效用调优建模为单步价值型强化学习:在给定请求上下文时,智能体选择一组效用权重向量以重加权排序器输出,最大化请求级参与度奖励。通过标量参数在推理时进行帕累托前沿扫掠,生成一组策略与经验帕累托前沿,作为运营策略选择的治理工具。PRL-PUTS与排序推理并行运行,不增加服务延迟。我们通过无偏探索日志进行离线分析,并在Pinterest首页进行在线实验,结果显示相比基线,成功会话率提升0.13%,是核心用户参与度指标。
原文摘要 · Abstract (English)
Large-scale recommenders encode multi-objective trade-offs by combining multiple predicted outcomes into a single utility score. Although this utility layer can be updated independently of the ranker, weight tuning remains largely manual, globally applied, slow to adapt to changing environments and business needs, and hard to govern as priorities shift. We propose PRL-PUTS, a Production-ready, ranker independent RL framework for Personalized Utility-weight Tuning with Pareto Sweeping. We cast utility tuning as a one-step, value-based RL problem: given request context, an agent selects a utility-weight vector that re-weights ranker predictions to maximize request-level engagement rewards. To visualize performance across the trade-off spectrum and allow decision makers to update the deployed operating policy instantly, we adopt an inference-time Pareto frontier sweeping via a scalarization parameter, producing a family of policies and an empirical Pareto frontier used as a governance artifact for operating policy selection. PRL-PUTS runs in parallel with ranking inference without adding serving latency. We validate PRL-PUTS with offline analysis using unbiased exploration logs and online experiments on Pinterest Homefeed where PRL-PUTS showed significant increases in engagement compared to baseline such as +0.13\% increase in successful session, a core metric for user engagement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。