用视觉语言嵌入减少强化学习中的专家反馈次数
Reducing Oracle Feedback with Vision-Language Embeddings for Preference-Based RL
- 用视觉语言模型生成初步偏好判断，只在不确定时才请求专家
- 实验显示专家提问量减少80%，且模型可跨任务泛化
- 适合需要大量标注但预算有限的机器人学习场景
基于偏好的强化学习可通过比较学习有效奖励函数，但其可扩展性受限于高昂的专家反馈成本。轻量级视觉-语言嵌入（VLE）模型提供更低成本的替代方案，但其噪声输出限制了其作为独立奖励生成器的效果。为此，我们提出ROVED，一种结合VLE监督与针对性专家反馈的混合框架。该方法利用VLE生成片段级偏好，并通过过滤机制识别高不确定性样本，仅对这些样本调用专家。此外，我们引入参数高效微调方法，基于获取的专家反馈逐步优化VLE，实现协同提升。该设计既保持嵌入模型的可扩展性，又确保专家反馈的准确性，避免两者效率短板。在多个机器人操作任务中，ROVED在匹配或超越现有方法的同时，将专家查询量降低高达80%。值得注意的是，优化后的VLE具备跨任务泛化能力，累计标注节省可达90%，凸显将可扩展嵌入与精准专家监督结合在偏好式强化学习中的实际价值。
原文摘要 · Abstract (English)
Preference-based reinforcement learning can learn effective reward functions from comparisons, but its scalability is constrained by the high cost of oracle feedback. Lightweight vision-language embedding (VLE) models provide a cheaper alternative, but their noisy outputs limit their effectiveness as standalone reward generators. To address this challenge, we propose ROVED, a hybrid framework that combines VLE-based supervision with targeted oracle feedback. Our method uses the VLE to generate segment-level preferences and defers to an oracle only for samples with high uncertainty, identified through a filtering mechanism. In addition, we introduce a parameter-efficient fine-tuning method that adapts the VLE with the obtained oracle feedback in order to improve the model over time in a synergistic fashion. This ensures the retention of the scalability of embeddings and the accuracy of oracles, while avoiding their inefficiencies. Across multiple robotic manipulation tasks, ROVED matches or surpasses prior preference-based methods while reducing oracle queries by up to 80%. Remarkably, the adapted VLE generalizes across tasks, yielding cumulative annotation savings of up to 90%, highlighting the practicality of combining scalable embeddings with precise oracle supervision for preference-based RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。