arXiv:2608.00289cs.AIcs.HC2026-08中稿 · ACM IMWUT

用无线电波识别人与物体互动,兼顾隐私与光照限制。

RF-HOI: Recognize Human-Object Interaction with Radio Frequency Signals

论文配图:RF-HOI: Recognize Human-Object Interaction with Radio Frequency Signals
图 1 · 摘自论文原文
  • 融合毫米波雷达与RFID信号实现动作与物品同步识别
  • 合成数据训练使系统在真实场景下性能显著提升
  • 适合需要隐私保护的智能交互场景

识别人与物体互动(HOI)对智能系统至关重要,支撑虚拟现实、具身智能和辅助机器人等应用。然而,基于视觉的方法存在隐私担忧和弱光环境下的性能下降问题。本文提出RF-HOI,首个仅依赖无线电频率(RF)信号实现HOI识别的框架。核心挑战在于单一模态的RF感知难以同时识别动作与交互对象。为此,RF-HOI创新性地融合毫米波雷达与RFID信号,实现动作识别与目标识别同步。另一挑战是跨多样场景的真实训练数据有限,影响模型泛化能力。为此,我们开发了一个模拟器,可大规模生成多模态RF数据,支持仅用少量真实数据进行微调。实验表明,RF-HOI优于所有基线方法,接近视觉模型表现;且多样合成数据能显著提升真实场景性能。结果验证了多模态RF感知在鲁棒、隐私友好的HOI识别中的潜力及数据合成的有效性。

原文摘要 · Abstract (English)

Recognizing Human-Object Interactions (HOI) is essential for intelligent systems, underpinning applications in virtual and augmented reality, embodied AI, and assistive robotics. However, vision-based HOI methods face challenges in privacy concerns and poor light conditions. In this work, we introduce RF-HOI, the first framework that only uses radio frequency (RF) signals for HOI recognition. A key challenge of RF-HOI is that single-modality RF sensing is insufficient to recognize both actions and the objects being interacted with. RF-HOI addresses this through a novel modality fusion that combines mmWave radar and RFID, enabling simultaneous action recognition and target identification. Another challenge is limited training data across diverse setups, which impairs the generalizability of the recognition model. To overcome this, we develop a simulator that synthesizes multimodal RF data for diverse HOIs at scale, allowing us to fine-tune with only a small amount of real-world data. Experiment results show that RF-HOI outperforms all baselines, approaching vision model performance, and that our diverse synthetic training data can significantly boost our system's performance on real-world scenarios. These results highlight the potential of multimodal RF sensing for robust and privacy-preserving HOI recognition as well as the effectiveness of our RF data synthesis.

人机交互无线感知多模态融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。