用真实社交互动数据训练大模型，无需人工标注奖励。
RLNVR: Reinforcement Learning from Non-Verified Real-World Rewards
- 通过基线归一化和语义相似性迁移奖励信号。
- 在Bluesky上优化内容生成，提升质量和训练稳定性。
- 适合缺乏人工标注的现实场景应用。
本文提出RLNVR（从非验证真实世界奖励中进行强化学习），一种利用噪声大的真实反馈信号训练语言模型的框架，无需人工验证。传统基于人类反馈的强化学习（RLHF）依赖昂贵且需验证的奖励信号，在多数现实场景中不切实际。RLNVR通过基线归一化和基于语义相似性的奖励迁移解决该问题。我们以Walter原型系统为例，使用Bluesky平台的真实社交互动数据优化社交媒体内容生成。实验表明，该方法显著提升了内容质量与训练稳定性，未来工作将开展全面评估。本文还结合GSPO（组序列策略优化）与可选的无监督环境设计（UED）课程，增强在噪声隐式奖励下的稳定性和多样性。据我们所知，这是首次在该应用背景下将GSPO风格的归一化与UED风格的课程相结合用于大模型内容生成，视为一种实用集成而非新算法。
原文摘要 · Abstract (English)
This paper introduces RLNVR (Reinforcement Learning from Non-Verified Rewards), a framework for training language models using noisy, real-world feedback signals without requiring explicit human verification. Traditional RLHF requires expensive, verified reward signals that are impractical in many real-world domains. RLNVR addresses this challenge through baseline normalization and semantic similarity-based reward transfer. We demonstrate RLNVR through Walter, a prototype system that optimizes social media content generation using actual engagement data from Bluesky. Our experimental results show significant improvements in content quality and training stability, with comprehensive evaluation planned for future work. Positioning: We present a practical framework that combines RLNVR with GSPO (Group Sequence Policy Optimization) and an optional UED (Unsupervised Environment Design) curriculum to improve stability and diversity under noisy, implicit rewards. To our knowledge, combining GSPO-style normalization with a UED-style curriculum for LLM content generation from implicit social engagement has not been previously documented in this applied setting; we frame this as an applied integration rather than a new algorithm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。