用多模态大模型自动设计并优化强化学习奖励函数，提升人机对齐效果。
VIRAL: Vision-grounded Integration for Reward design And Learning
- 通过多模态LLM自动生成与迭代优化奖励函数
- 在5个Gymnasium环境中加速新行为学习并更好匹配用户意图
- 支持人类反馈或视频描述引导，适合交互式智能系统研发
当前人工智能中人机对齐仍是关键挑战。强化学习依赖奖励函数，而设计不当的奖励易引发风险。近期研究表明，利用大语言模型生成奖励可超越人类表现。我们提出VIRAL，一种基于多模态大语言模型的奖励生成与优化流水线。VIRAL能根据给定环境和目标提示或标注图像，自主生成并交互式改进奖励函数。优化过程可结合人类反馈，或由视频大模型生成的策略描述进行引导，解释代理行为。我们在五个Gymnasium环境中评估了VIRAL，结果表明其能加速新行为学习，并显著提升与用户意图的一致性。源代码与演示视频见：https://github.com/VIRAL-UCBL1/VIRAL 及 https://youtu.be/Hqo82CxVT38。
原文摘要 · Abstract (English)
The alignment between humans and machines is a critical challenge in artificial intelligence today. Reinforcement learning, which aims to maximize a reward function, is particularly vulnerable to the risks associated with poorly designed reward functions. Recent advancements has shown that Large Language Models (LLMs) for reward generation can outperform human performance in this context. We introduce VIRAL, a pipeline for generating and refining reward functions through the use of multi-modal LLMs. VIRAL autonomously creates and interactively improves reward functions based on a given environment and a goal prompt or annotated image. The refinement process can incorporate human feedback or be guided by a description generated by a video LLM, which explains the agent's policy in video form. We evaluated VIRAL in five Gymnasium environments, demonstrating that it accelerates the learning of new behaviors while ensuring improved alignment with user intent. The source-code and demo video are available at: https://github.com/VIRAL-UCBL1/VIRAL and https://youtu.be/Hqo82CxVT38.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。