arXiv:2602.04278cs.IR2026-02被引 1

用奖励信号筛选数据,让大模型推荐更省力高效。

MiniRec: Data-Efficient Reinforcement Learning for LLM-based Recommendation

  • 根据奖励高低和梯度方向筛选样本,贴合强化学习真实训练过程。
  • 仅用10%数据训练,推荐效果损失小于5%,显著降低算力开销。
  • 适合资源有限但需高精度推荐的场景,如小团队部署大模型系统。

将强化学习(RL)融入大语言模型(LLM)为推荐系统带来新机遇,可激发推理能力并提升用户偏好建模。然而,基于RL的LLM推荐面临显著效率挑战,全量数据训练成本高昂。现有数据选择方法依赖可学习性或代表性,其基于损失或梯度、或数据覆盖度的标准常与RL学习动态错位,导致性能不佳。为此,我们提出MiniRec,一个专为基于强化学习的LLM推荐设计的数据选择框架。MiniRec利用关键的强化学习信号——奖励,评估样本可学习性:剔除奖励过高(太容易)或过低(太难)的样本;通过对比样本梯度与近似“理想”全局优化轨迹,衡量代表性,选择主要驱动模型更新的样本,并引入多样性机制减少冗余。结合从易到难的课程学习策略,MiniRec大幅降低训练成本,同时基本保持性能。大量实验验证了其有效性,凸显奖励对齐、轨迹感知的数据选择在基于强化学习的LLM推荐中的重要性。

原文摘要 · Abstract (English)

The integration of reinforcement learning (RL) into large language models (LLMs) has opened new opportunities for recommender systems by eliciting reasoning and improving user preference modeling. However, RL-based LLM recommendation faces significant efficiency challenges, making full-data training costly. Existing data selection methods define sample value based on learnability or representativeness, yet their loss- or gradient-driven or dataset coverage-driven criteria often misalign with RL learning dynamics, resulting in suboptimal performance. To address this, we propose MiniRec, a data selection framework tailored for RL-based LLM recommendation. MiniRec evaluates sample learnability using key RL signals -- rewards -- pruning samples that are too easy (too high reward) or too difficult (consistently low reward). It assesses representativeness by aligning sample gradients with the approximated "ideal" global RL optimization trajectory, selecting samples that mainly drive model updates, and it also enforces diversity to reduce redundancy. Combined with a curriculum learning strategy from easy to hard samples, MiniRec significantly reduces training cost while largely preserving performance. Extensive experiments demonstrate MiniRec's effectiveness, highlighting the importance of reward-aligned, trajectory-informed data selection in RL-based LLM recommendation.

强化学习推荐系统大模型数据高效

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。