用测试时微调让旧数据更贴合当前目标,显著提升机器人导航与操作性能。
Test-time Offline Reinforcement Learning on Goal-related Experience
- 根据当前状态和目标相关性筛选旧数据,自监督选择高质量经验
- 仅用少量梯度步数微调,即在高维任务中实现显著性能提升
- 适合需要快速适应新任务的机器人系统,尤其计算资源有限场景
基础模型将大量信息压缩进单一大型神经网络,可针对特定任务进行查询。这一框架与离线目标条件强化学习高度相似:通用价值函数在大量目标上训练,而策略在每个测试回合中仅评估单个目标。研究表明,通过测试时训练可显著提升基础模型性能,使其适应当前任务。我们发现,在测试时对与当前目标相关的离线经验进行强化学习,可在较低计算成本下大幅提升策略表现。提出一种新型自监督数据选择准则,根据状态相关性和目标达成质量从离线数据集中筛选过渡样本。在多种高维运动导航与操作任务中验证,仅用数步梯度更新微调策略,即可显著优于标准离线预训练。所提目标条件测试时训练(GC-TTT)算法以滚动视野方式在评估期间持续优化策略,随轨迹推进动态适应。最后研究推理阶段计算分配,表明在相同计算开销下,GC-TTT带来的性能增益无法通过单纯扩大模型规模实现。
原文摘要 · Abstract (English)
Foundation models compress a large amount of information in a single, large neural network, which can then be queried for individual tasks. There are strong parallels between this widespread framework and offline goal-conditioned reinforcement learning algorithms: a universal value function is trained on a large number of goals, and the policy is evaluated on a single goal in each test episode. Extensive research in foundation models has shown that performance can be substantially improved through test-time training, specializing the model to the current goal. We find similarly that test-time offline reinforcement learning on experience related to the test goal can lead to substantially better policies at modest compute costs. We propose a novel self-supervised data selection criterion, which selects transitions from an offline dataset according to their relevance to the current state and quality with respect to the evaluation goal. We demonstrate across a wide range of high-dimensional loco-navigation and manipulation tasks that fine-tuning a policy on the selected data for a few gradient steps leads to significant performance gains over standard offline pre-training. Our goal-conditioned test-time training (GC-TTT) algorithm applies this routine in a receding-horizon fashion during evaluation, adapting the policy to the current trajectory as it is being rolled out. Finally, we study compute allocation at inference, demonstrating that, at comparable costs, GC-TTT induces performance gains that are not achievable by scaling model size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。