让机器人在缺少视觉或声音时仍能准确识别动作
Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalities
- 用多模态知识蒸馏，让模型学会利用部分可用信息
- 缺失模态时准确率下降不到10%，远优于传统方法
- 适合部署在传感器不全的机器人上，计算量减半
第一人称动作识别使机器人能辅助人机交互并监控任务进展。现有方法多仅依赖RGB视频，尽管音频等额外模态可在复杂条件下提升精度。然而，多数多模态方法假设所有模态在推理时均可用，一旦输入缺失，性能显著下降甚至失效。为此，我们提出KARMMA——一种针对第一人称动作识别、对缺失模态鲁棒的多模态知识蒸馏框架，训练和推理无需跨样本模态对齐。KARMMA将多模态教师模型的知识蒸馏到多模态学生模型中，该学生模型可利用所有可用模态，同时对缺失模态保持鲁棒性，支持在不同传感器配置下部署而无需重新训练。学生模型计算资源仅为教师的一半，轻量快速，适合机器人端部署。在Epic-Kitchens和Something-Something数据集上的实验表明，该学生模型在正常条件下达到竞争力精度，并在模态缺失情况下显著降低性能下降。
原文摘要 · Abstract (English)
Egocentric action recognition enables robots to facilitate human-robot interactions and monitor task progress. Existing methods often rely solely on RGB videos, although additional modalities, such as audio, can improve accuracy under challenging conditions. However, most multimodal approaches assume that all modalities are available at inference time, leading to significant accuracy drops, or even failure, when inputs are missing. To address this limitation, we introduce KARMMA, a multimodal Knowledge distillation framework for egocentric Action Recognition robust to Missing ModAlities that does not require modality alignment across all samples during training or inference. KARMMA distills knowledge from a multimodal teacher into a multimodal student that leverages all available modalities while remaining robust to missing ones, enabling deployment across diverse sensor configurations without retraining. Our student uses approximately 50% fewer computational resources than the teacher, resulting in a lightweight and fast model that is well suited for on-robot deployment. Experiments on Epic-Kitchens and Something-Something demonstrate that our student achieves competitive accuracy while significantly reducing performance degradation under missing modality conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。