让大模型直接学口语反馈，不靠打分也能进步
Language Models Can Learn from Verbal Feedback Without Scalar Rewards
- 用口语反馈当条件，直接训练模型生成响应
- 在线自举提升性能，比传统打分方法更有效
- 适合想用自然语言优化模型的开发者
大型语言模型通常通过人类或人工智能反馈进行强化学习训练，但这些方法常将复杂的反馈压缩为标量奖励，丢失丰富信息并引发尺度失衡。本文提出将口语反馈视为条件信号，受文本到图像生成中语言先验的启发，引入反馈条件策略（FCP）。FCP 直接从响应-反馈对中学习，通过最大似然训练近似反馈条件后验分布。我们进一步设计在线自举阶段：模型在正向条件下生成，并接收新反馈以自我优化。这种方法将反馈驱动学习重新定义为条件生成，而非奖励优化，使大模型能更充分地利用口语反馈。代码已开源。
原文摘要 · Abstract (English)
LLMs are often trained with RL from human or AI feedback, yet such methods typically compress nuanced feedback into scalar rewards, discarding much of their richness and inducing scale imbalance. We propose treating verbal feedback as a conditioning signal. Inspired by language priors in text-to-image generation, which enable novel outputs from unseen prompts, we introduce the feedback-conditional policy (FCP). FCP learns directly from response-feedback pairs, approximating the feedback-conditional posterior through maximum likelihood training on offline data. We further develop an online bootstrapping stage where the policy generates under positive conditions and receives fresh feedback to refine itself. This reframes feedback-driven learning as conditional generation rather than reward optimization, offering a more expressive way for LLMs to directly learn from verbal feedback. Our code is available at https://github.com/sail-sg/feedback-conditional-policy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。