用语言描述自动生成奖励,让智能体学会按自然语言控制流体。
LinguaFluid: Language Guided Fluid Control via Semantic Rewards in Reinforcement Learning
- 用SBERT计算语义相似度作为奖励信号,替代人工设计奖励函数。
- 在多个环境中实现与人工奖励相当的控制效果,无需手动调参。
- 适合希望用自然语言指导物理模拟或机器人控制的研究者。
在科学机器学习领域,强化学习中的奖励函数设计仍具挑战性,尤其当任务目标难以数值化时。现有方法多依赖启发式规则、人工工程或特定任务调参。本文提出一种语义对齐的强化学习方法:通过Sentence-Bidirectional Encoder Representations from Transformers (SBERT) 将当前状态与目标语义指令对齐,以计算奖励。策略基于目标文本描述与当前状态语义描述间的余弦相似度获得反馈,而非依赖人工定义的奖励函数。我们在多个环境上评估该方法,结果表明,即使无手工奖励函数,语义奖励仍可引导学习达到具有竞争力的控制行为。研究揭示了语言嵌入空间与传统欧氏空间之间的相关性。该框架为将智能体行为与自然语言目标对齐开辟新路径,并为大语言模型(LLMs)与流体控制应用的无缝集成奠定基础。
原文摘要 · Abstract (English)
In the domain of scientific machine learning, designing effective reward functions remains a challenge in reinforcement learning (RL), particularly in environments where task goals are difficult to specify numerically. Reward functions in existing work are predominantly based on heuristics, manual engineering, or task-specific tuning. In this work, we introduce a semantically aligned reinforcement learning method where rewards are computed by aligning the current state with a target semantic instruction using a Sentence-Bidirectional Encoder Representations from Transformers (SBERT). Instead of relying on manually defined reward functions, the policy receives feedback based on the reward, which is a cosine similarity between the goal textual description and the statement description in the episode. We evaluated our approach in several environments and showed that semantic reward can guide learning to achieve competitive control behavior, even in the absence of hand-crafted reward functions. Our study demonstrates a correlation between the language embedding space and the conventional Euclidean space. This framework opens new horizons for aligning agent behavior with natural language goals and lays the groundwork for a more seamless integration of larger language models (LLMs) and fluid control applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。