用手势控制声悬浮机器人，实现触觉音频和悬浮的无缝切换。
A Gesture-Based Visual Learning Model for Acoustophoretic Interactions using a Swarm of AcoustoBots
- 通过视觉语言模型识别手势并映射到多模态输出
- 最大数据集下识别准确率达98%，系统整体切换准确率87.8%
- 适合需要无接触交互的机器人操控场景
AcoustoBots是可移动的声学微粒机器人，能实现空中触觉、定向音频和声悬浮，但现有系统依赖预设指令，缺乏直观实时的人机控制界面。本文提出一种基于手势的视觉学习框架，用于与多模态AcoustoBot集群进行非接触式人机交互。系统结合ESP32-CAM手势捕捉、PhaseSpace运动追踪、集中式处理以及基于OpenCLIP的视觉学习模型（VLM）与线性探测，分类三种手部手势，并将其映射至触觉、音频和悬浮模态。验证准确率从约67%（小数据集）提升至接近98%（最大数据集）。在双AcoustoBot集成实验中，系统在90次测试中实现了87.8%的整体手势-模态切换准确率，端到端平均延迟为3.95秒。结果表明，基于视觉语言模型的手势接口在多模态人机集群交互中具有可行性。尽管当前系统受限于集中式处理、静态手势集及受控环境评估，但仍为更丰富、可扩展且易用的集群机器人交互奠定了基础。
原文摘要 · Abstract (English)
AcoustoBots are mobile acoustophoretic robots capable of delivering mid-air haptics, directional audio, and acoustic levitation, but existing implementations rely on scripted commands and lack an intuitive interface for real-time human control. This work presents a gesture-based visual learning framework for contactless human-swarm interaction with a multimodal AcoustoBot platform. The system combines ESP32-CAM gesture capture, PhaseSpace motion tracking, centralized processing, and an OpenCLIP-based visual learning model (VLM) with linear probing to classify three hand gestures and map them to haptics, audio, and levitation modalities. Validation accuracy improved from about 67% with a small dataset to nearly 98% with the largest dataset. In integrated experiments with two AcoustoBots, the system achieved an overall gesture-to-modality switching accuracy of 87.8% across 90 trials, with an average end-to-end latency of 3.95 seconds. These results demonstrate the feasibility of using a vision-language-model-based gesture interface for multimodal human-swarm interaction. While the current system is limited by centralized processing, a static gesture set, and controlled-environment evaluation, it establishes a foundation for more expressive, scalable, and accessible swarm robotic interfaces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。