用少量力觉数据让视觉机器人模型学会触觉，还能保持原有能力。
Multisensory Continual Learning: Adapting Pretrained Visuomotor Policies to Force
- 通过多阶段融合与未来预测，将力觉信号融入预训练视觉策略
- 在真实接触任务中表现优异，且原任务性能不降反升
- 适合需快速适配新传感器的机器人持续学习场景
机器人操作常依赖视觉之外的感官反馈，尤其在接触密集场景中，力、触觉或音频信号能揭示图像无法直接观察的交互状态。然而，这些模态通常依赖特定硬件和任务，大规模多感官机器人数据集仍稀缺。因此，为所有可能传感器预训练策略不现实。本文研究多感官持续学习：在不丢失原有传感器性能的前提下，将预训练的视觉策略适配到引入新模态的新任务。提出多感官世界模型（MuSe），通过多阶段融合、多感官未来预测及预训练数据的经验回放，将有限的多感官数据融入视觉仅模型。以力-扭矩传感为例，对预训练视觉策略进行增强，并在真实世界操作任务上评估。实验表明，MuSe在接触密集微调任务中表现良好，同时在原始预训练任务上性能保持甚至提升。结果表明，少量多感官数据即可扩展机器人在分布外任务中的通用能力。项目网站：https://jadenvc.github.io/multisensory-continual-learning/
原文摘要 · Abstract (English)
Robot manipulation often relies on sensory feedback beyond vision, particularly in contact-rich settings where force, tactile, or audio signals reveal interaction states that are not directly observable from images. However, these modalities are often hardware- and task-specific, and large-scale multisensory robot datasets remain scarce. As a result, it is impractical to pretrain policies with every sensor they may encounter. We study multisensory continual learning: adapting a pretrained robot policy to new tasks with newly introduced modalities while preserving performance under the original sensor suite. We propose MultiSensory World Model (MuSe), which incorporates limited multisensory data into pretrained vision-only policies through multi-stage fusion, multisensory future prediction, and experience replay over pretraining data. We instantiate MuSe by augmenting a pretrained vision-only policy with force-torque sensing and evaluate it on real-world manipulation tasks. Our experiments show that MuSe performs strongly on contact-rich finetuning tasks while preserving, and in some cases improving, performance on the original pretraining tasks. These results suggest that a modest multisensory dataset can improve general robot capabilities beyond the finetuning distribution. Project website: https://jadenvc.github.io/multisensory-continual-learning/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。