用表格型基础模型提升机器人策略学习的探索效率。
Can Tabular Foundation Models Guide Exploration in Robot Policy Learning?
- 结合局部更新与全局搜索，通过SVD构建低维策略空间。
- 利用预训练表格模型在有限采样下筛选候选策略，加速收敛。
- 适合追求样本高效、少依赖初始值的机器人控制研究者。
机器人高维连续控制中的策略优化仍具挑战性。主流方法多为局部搜索，需大量调参和良好初始值；而更全局的方法通常采样成本过高。本文提出TFM-S3，一种基于表格型基础模型的混合式局部-全局方法，在有限采样预算下提升探索效率。该方法高频进行局部更新，周期性执行全局搜索：每次搜索中，通过SVD动态构建低维策略子空间，并在此空间内迭代优化；同时利用预训练表格基础模型，从少量上下文预测候选策略回报，实现大规模策略筛选。在连续控制基准测试中，TFM-S3在相同采样预算下显著加速早期收敛，并优于TD3及种群基线方法的最终性能。结果表明，基础模型是实现样本高效机器人连续控制策略学习的强大新工具。
原文摘要 · Abstract (English)
Policy optimization in high-dimensional continuous control for robotics remains a challenging problem. Predominant methods are inherently local and often require extensive tuning and carefully chosen initial guesses for good performance, whereas more global and less initialization-sensitive search methods typically incur high rollout costs. We propose TFM-S3, a tabular hybrid local-global method for improving global exploration in robot policy learning with limited rollout cost. We interleave high-frequency local updates with intermittent rounds of global search. In each search round, we construct a dynamically updated low-dimensional policy subspace via SVD and perform iterative surrogate-guided refinement within this space. A pretrained tabular foundation model predicts candidate returns from a small context set, enabling large-scale screening with limited rollout cost. Experiments on continuous control benchmarks show that TFM-S3 consistently accelerates early-stage convergence and improves final performance compared to TD3 and population-based baselines under an identical rollout budget. These results demonstrate that foundation models are a powerful new tool for creating sample-efficient policy learning methods for continuous control in robotics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。