用语言嵌入生成隐式奖励,让机器人更高效学会复杂任务
Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning
- 将任务描述转为语义化进度信号,自动提供持续反馈
- 训练速度更快,成功率更高,解决传统方法无法完成的任务
- 适合需要快速学习、少人工设计的智能体应用
我们提出 Reward-Zero,一种通用的隐式奖励机制,将自然语言任务描述转化为稠密、语义对齐的进展信号,用于强化学习(RL)。该机制通过比较任务规范的嵌入与智能体交互经验的嵌入,生成连续的完成感信号。它无需任务定制化设计,即可补充稀疏或延迟的环境反馈。集成到标准RL框架后,能加速探索、稳定训练并提升跨任务泛化能力。实验证明,使用 Reward-Zero 的智能体收敛更快、最终成功率更高,甚至在部分复杂任务中成功解决了人工设计奖励无法处理的问题。此外,我们构建了一个小型基准,用于评估执行过程中基于语言嵌入的完成感。结果表明,语言驱动的隐式奖励函数是实现更高效、可扩展、通用强化学习的可行路径。代码将在同行评审后发布。
原文摘要 · Abstract (English)
We introduce Reward-Zero, a general-purpose implicit reward mechanism that transforms natural-language task descriptions into dense, semantically grounded progress signals for reinforcement learning (RL). Reward-Zero serves as a simple yet sophisticated universal reward function that leverages language embeddings for efficient RL training. By comparing the embedding of a task specification with embeddings derived from an agent's interaction experience, Reward-Zero produces a continuous, semantically aligned sense-of-completion signal. This reward supplements sparse or delayed environmental feedback without requiring task-specific engineering. When integrated into standard RL frameworks, it accelerates exploration, stabilizes training, and enhances generalization across diverse tasks. Empirically, agents trained with Reward-Zero converge faster and achieve higher final success rates than conventional methods such as PPO with common reward-shaping baselines, successfully solving tasks that hand-designed rewards could not in some complex tasks. In addition, we develop a mini benchmark for the evaluation of completion sense during task execution via language embeddings. These results highlight the promise of language-driven implicit reward functions as a practical path toward more sample-efficient, generalizable, and scalable RL for embodied agents. Code will be released after peer review.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。