用语言模型从网络视频直接学奖励函数，省去繁琐数据处理。
Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos
- 上层用视觉语言模型对比行为与专家视频，生成反馈
- 下层用大语言模型将反馈转为奖励更新，形成双向优化
- 无需定制数据流程，适合复杂行为合成的快速奖励设计
从示范中学习，尤其是来自人类和动物等生物专家的示范，常面临数据获取难题。尽管近期方法利用互联网视频进行学习，但仍需复杂的、任务特定的数据提取与重定向流程。本文提出一种语言模型辅助的双层规划框架，使强化学习智能体可直接从互联网视频中学习奖励函数，跳过专门的数据准备环节。该框架包含两个层次：上层由视觉-语言模型（VLM）通过对比学习者行为与专家视频提供反馈；下层由大型语言模型（LLM）将反馈转化为奖励更新。VLM与LLM在此双层结构中协作，采用“链式法则”推导出有效的奖励学习搜索方向。我们在YouTube视频上验证了该方法，结果表明，该方法能高效地从生物专家视频中学习奖励，实现复杂行为的合成。
原文摘要 · Abstract (English)
Learning from Demonstrations, particularly from biological experts like humans and animals, often encounters significant data acquisition challenges. While recent approaches leverage internet videos for learning, they require complex, task-specific pipelines to extract and retarget motion data for the agent. In this work, we introduce a language-model-assisted bi-level programming framework that enables a reinforcement learning agent to directly learn its reward from internet videos, bypassing dedicated data preparation. The framework includes two levels: an upper level where a vision-language model (VLM) provides feedback by comparing the learner's behavior with expert videos, and a lower level where a large language model (LLM) translates this feedback into reward updates. The VLM and LLM collaborate within this bi-level framework, using a "chain rule" approach to derive a valid search direction for reward learning. We validate the method for reward learning from YouTube videos, and the results have shown that the proposed method enables efficient reward design from expert videos of biological agents for complex behavior synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。