用普通摄像头实现无传感器空中写字,支持中文全字符集。
Finger in Camera Speaks Everything: Unconstrained Air-Writing for Real-World
- 仅用RGB摄像头捕捉指尖轨迹,无需雷达或脑电设备
- 构建包含880万帧的10万级中文空中写字数据集
- 适合手机、电脑等日常设备,推动真实场景人机交互
空中写字融合计算机视觉与自然语言处理,是直观的人机交互方式。当前方法依赖雷达、脑电等复杂传感器,且缺乏覆盖全面字库的视频数据集,限制了实际应用。为此,我们提出首个基于视频的中文空中写字数据集AWCV-100K-UCAS2024,使用普通RGB相机在真实场景中采集,包含880万帧视频,涵盖GB2312-80一级字符集全部3,755个汉字。同时提出视频字符识别模型VCRec,从稀疏视觉线索中提取指尖特征,并通过时空序列模块分析。实验表明,VCRec在定量和定性上均优于现有模型,显著提升识别性能。该方案仅需低成本摄像头,适用于手机、笔记本等广泛场景。代码与数据将公开于https://github.com/wmeiqi/AWCV。
原文摘要 · Abstract (English)
Air-writing is a challenging task that combines the fields of computer vision and natural language processing, offering an intuitive and natural approach for human-computer interaction. However, current air-writing solutions face two primary challenges: (1) their dependency on complex sensors (e.g., Radar, EEGs and others) for capturing precise handwritten trajectories, and (2) the absence of a video-based air-writing dataset that covers a comprehensive vocabulary range. These limitations impede their practicality in various real-world scenarios, including the use on devices like iPhones and laptops. To tackle these challenges, we present the groundbreaking air-writing Chinese character video dataset (AWCV-100K-UCAS2024), serving as a pioneering benchmark for video-based air-writing. This dataset captures handwritten trajectories in various real-world scenarios using commonly accessible RGB cameras, eliminating the need for complex sensors. AWCV-100K-UCAS2024 includes 8.8 million video frames, encompassing the complete set of 3,755 characters from the GB2312-80 level-1 set (GB1). Furthermore, we introduce our baseline approach, the video-based character recognizer (VCRec). VCRec adeptly extracts fingertip features from sparse visual cues and employs a spatio-temporal sequence module for analysis. Experimental results showcase the superior performance of VCRec compared to existing models in recognizing air-written characters, both quantitatively and qualitatively. This breakthrough paves the way for enhanced human-computer interaction in real-world contexts. Moreover, our approach leverages affordable RGB cameras, enabling its applicability in a diverse range of scenarios. The code and data examples will be made public at https://github.com/wmeiqi/AWCV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。