用大模型知识蒸馏，让扫地机器人更聪明地识别并避开或吸走物品。
VLM-Vac: Enhancing Smart Vacuums through VLM Knowledge Distillation and Language-Guided Experience Replay
- 用视觉语言模型做零样本物体识别，指导小模型学习
- 蒸馏后的小模型查询次数减少90%以上，仍保持高准确率
- 基于语言提示的回放机制，适合家庭环境持续学习
本文提出VLM-Vac，一种增强智能扫地机器人自主性的新框架。该方法结合视觉语言模型（VLM）的零样本物体检测能力与知识蒸馏（KD）策略。通过VLM，机器人可在多样背景中将物体分类为可避让或可吸入类别。然而频繁调用VLM计算开销大，难以部署。为此，我们采用KD过程，逐步将VLM的核心知识迁移到更小、更高效的模型中。真实场景实验表明，该小模型能持续从VLM学习，随时间推移显著减少查询次数。此外，针对动态家庭环境中的持续学习挑战，我们提出基于语言引导采样的新颖经验回放方法。结果表明，该方法不仅节能，且在复杂背景下对小物体的检测性能优于传统视觉聚类方法。
原文摘要 · Abstract (English)
In this paper, we propose VLM-Vac, a novel framework designed to enhance the autonomy of smart robot vacuum cleaners. Our approach integrates the zero-shot object detection capabilities of a Vision-Language Model (VLM) with a Knowledge Distillation (KD) strategy. By leveraging the VLM, the robot can categorize objects into actionable classes -- either to avoid or to suck -- across diverse backgrounds. However, frequently querying the VLM is computationally expensive and impractical for real-world deployment. To address this issue, we implement a KD process that gradually transfers the essential knowledge of the VLM to a smaller, more efficient model. Our real-world experiments demonstrate that this smaller model progressively learns from the VLM and requires significantly fewer queries over time. Additionally, we tackle the challenge of continual learning in dynamic home environments by exploiting a novel experience replay method based on language-guided sampling. Our results show that this approach is not only energy-efficient but also surpasses conventional vision-based clustering methods, particularly in detecting small objects across diverse backgrounds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。