分三阶段渐进对齐推荐基础模型，提升业务指标表现
Progressive Alignment of Recommender Foundation Model through Multi-Phase Post-Training
- 先线性探针稳定下游头，再全量微调优化任务，最后强化学习对齐业务目标
- 线上实验显示新框架显著优于单阶段方法，业务指标提升明显
- 适合追求推荐系统实际效果的工业级研发团队使用
推荐领域基础模型展现出建模长期用户行为的强大能力。实践中，通常通过监督微调（SFT）将预训练基础模型适配到多种下游服务场景。然而，优化点击或点赞等任务目标，并不保证推荐策略与决定质量的业务指标对齐。本文提出一种三阶段渐进后训练框架，显式分离下游适配与业务指标对齐过程：第一阶段为线性探针（LP），在冻结预训练表征空间中稳定随机初始化的下游头；第二阶段为全量微调（FFT），联合优化完整模型以适应目标任务；第三阶段为强化学习微调（RFT），基于学习的奖励模型对齐实际业务目标。不直接在稀疏的业务目标上优化服务策略，而是用密集隐式反馈训练策略，仅用业务指标监督用于奖励建模。离线实验表明，该渐进式 LP-FFT-RFT 框架优于单阶段方案，且基于奖励的对齐比直接使用奖励模型排序更有效。大规模线上 A/B 测试进一步验证，该框架相比传统非基础模型基线显著提升了生产环境推荐质量。参考实现已开源：https://github.com/webtoon/rec-fm-progressive-alignment
原文摘要 · Abstract (English)
Foundation model(FM) for recommendation has shown strong ability to model long-horizon sequential user behavior. In practice, a single pretrained foundation model is often adapted to diverse downstream serving surfaces through Supervised Fine-Tuning(SFT). However, optimizing task-specific objectives such as clicks or likes does not necessarily align the serving policy with the business metrics that determine recommendation quality. We propose a three-phase progressive post-training framework that explicitly separates downstream adaptation from business-metric alignment. The adaptation stage is decomposed into Linear Probing(LP) and Full Fine-Tuning(FFT): LP first stabilizes randomly initialized downstream heads within a frozen pretrained representation space, and FFT then jointly specializes the full model for the target task. On top of this stabilized policy, Reinforcement Fine-Tuning(RFT) aligns the model with practical business objectives using a learned reward model. Rather than directly optimizing the serving policy on sparse business targets, we train the policy on dense implicit feedback and use business-metric supervision only for reward modeling. Offline experiments show that the progressive LP-FFT-RFT framework outperforms single-phase alternatives, and that reward-based alignment yields a stronger serving policy than directly using the reward model itself for ranking. Large-scale online A/B tests further show that the proposed framework improves production recommendation quality over a conventional non-foundation baseline. A reference implementation is available at https://github.com/webtoon/rec-fm-progressive-alignment
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。