用语言反馈让模仿学习模型从错误示范中汲取经验,提升泛化能力。
FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Tasks
- 将语言反馈嵌入Transformer输入,指导模型理解最优与非最优行为差异。
- 在自定义环境上实现显著更强的组合泛化与鲁棒性,数据效率更高。
- 适合需要语言指导的具身视觉-语言任务,替代传统奖励机制。
当前具身AI方法多依赖专家示范来学习策略,但缺乏对示范动作质量的评估机制,通常只能学习最优行为,或复制错误与低效。强化学习虽可缓解此问题,但探索过程牺牲了数据效率。本文探讨如何在模仿学习中利用建设性语言反馈,使模型从最优与非最优示范中学习稳健表征。我们直接将语言反馈嵌入Transformer策略的输入序列,并可选地引入辅助自监督学习目标以预测反馈。在自研的BabyAI-XGen环境中测试表明,该方法显著提升了智能体在多种具身视觉-语言任务中的组合泛化能力和鲁棒性,证明语言反馈是数据高效且直观的替代方案,能有效将非最优行为转化为学习机会。结果表明,语言反馈是语言指定具身任务中极具竞争力的替代中间标量奖励的方式。
原文摘要 · Abstract (English)
Current approaches to embodied AI tend to learn policies from expert demonstrations. However, without a mechanism to evaluate the quality of demonstrated actions, they are limited to learning from optimal behaviour, or they risk replicating errors and inefficiencies. While reinforcement learning offers one alternative, the associated exploration typically results in sacrificing data efficiency. This work explores how agents trained with imitation learning can learn robust representations from both optimal and suboptimal demonstrations when given access to constructive language feedback as a means to contextualise different modes of behaviour. We directly provide language feedback embeddings as part of the input sequence into a Transformer-based policy, and optionally complement the traditional next action prediction objective with auxiliary self-supervised learning objectives for feedback prediction. We test our approach on a range of embodied Vision-and-Language tasks in our custom BabyAI-XGen environment and show significant improvements in agents' compositional generalisation abilities and robustness, suggesting that our data-efficient method allows models to successfully convert suboptimal behaviour into learning opportunities. Overall, our results suggest that language feedback is a competitive and intuitive alternative to intermediate scalar rewards for language-specified embodied tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。