arXiv:2605.03408cs.LGcs.AI2026-05

用大模型自动设计强化学习的观测与奖励接口,减少人工干预。

Discovering Reinforcement Learning Interfaces with Large Language Models

论文配图:Discovering Reinforcement Learning Interfaces with Large Language Models
图 1 · 摘自论文原文
  • 通过大模型生成可执行代码,进化优化观测和奖励函数联合方案。
  • 仅凭成功轨迹指标,就能在多个任务上发现有效接口,单优化任一组件会失败。
  • 适合需要快速构建新强化学习任务的开发者或研究者使用。

强化学习系统依赖于环境接口,包括观测映射与奖励函数,但为新任务构建这些接口通常需大量手动工作。尽管近期工作已用大语言模型(LLMs)自动化奖励设计,但这些方法假设观测固定,未解决完整任务接口合成问题。本文研究从原始模拟器状态中发现强化学习任务接口,需同时生成观测映射与奖励函数。提出LIMEN(代码见https://github.com/Lossfunk/LIMEN),一个由大模型引导的进化框架,将候选接口生成为可执行程序,并利用策略训练反馈迭代优化。在多个新型离散网格世界任务及连续控制领域(涵盖运动与操作)中,观测与奖励的联合演化能仅凭轨迹级成功指标发现有效接口,而单独优化任一组件至少在一个领域失效。结果表明,从原始状态自动构建强化学习接口可显著减少人工工程量,且观测与奖励常需协同设计;单一组件优化在评估套件中至少一个领域会灾难性失败。

原文摘要 · Abstract (English)

Reinforcement learning systems rely on environment interfaces that specify observations and reward functions, yet constructing these interfaces for new tasks often requires substantial manual effort. While recent work has automated reward design using large language models (LLMs), these approaches assume fixed observations and do not address the broader challenge of synthesizing complete task interfaces. We study RL task interface discovery from raw simulator state, where both observation mappings and reward functions must be generated. We propose LIMEN (Code available at https://github.com/Lossfunk/LIMEN), a LLM guided evolutionary framework that produces candidate interfaces as executable programs and iteratively refines them using policy training feedback. Across novel discrete gridworld tasks and continuous control domains spanning locomotion and manipulation, joint evolution of observations and rewards discovers effective interfaces given only a trajectory-level success metric, while optimizing either component alone fails on at least one domain. These results demonstrate that automatic construction of RL interfaces from raw state can substantially reduce manual engineering and that observation and reward components often benefit from co-design, as single-component optimization fails catastrophically on at least one domain in our evaluation suite.

强化学习大模型接口设计自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。