用事件流辅助摄像头提升手语识别准确率,解决模糊与光照问题。
Sign Language Translation using Frame and Event Stream: Benchmark Dataset and Algorithms
- 结合RGB图像与事件流数据,捕捉更清晰的手势动态。
- 构建含1.5万样本的中文手语数据集VECSL,覆盖2568个汉字。
- 提出M²-SLT框架,实现细粒度与粗粒度手势联合识别,性能领先。
精准的手语理解对残障人士沟通至关重要。当前手语翻译算法多依赖RGB帧,受限于固定帧率、光照变化及快速动作导致的运动模糊。受事件相机在其他领域成功应用的启发,本文提出利用事件流辅助RGB相机采集手势数据,以应对上述挑战。首先,使用DVS346相机采集大规模的RGB-事件手语数据集VECSL,包含15,676组RGB-事件样本、15,191个词素(glosses),涵盖2,568个中文字符。数据采集覆盖多种室内外环境、多视角、不同光照强度及相机运动。由于该任务尚无基准算法,我们重新训练并评估了多个前沿手语翻译模型,认为该基准可有效支持后续研究。此外,提出新型RGB-事件手语翻译框架M²-SLT,融合细粒度微手势与粗粒度宏手势检索,在所提数据集上达到当前最优性能。源代码与数据集将公开于https://github.com/Event-AHU/OpenESL。
原文摘要 · Abstract (English)
Accurate sign language understanding serves as a crucial communication channel for individuals with disabilities. Current sign language translation algorithms predominantly rely on RGB frames, which may be limited by fixed frame rates, variable lighting conditions, and motion blur caused by rapid hand movements. Inspired by the recent successful application of event cameras in other fields, we propose to leverage event streams to assist RGB cameras in capturing gesture data, addressing the various challenges mentioned above. Specifically, we first collect a large-scale RGB-Event sign language translation dataset using the DVS346 camera, termed VECSL, which contains 15,676 RGB-Event samples, 15,191 glosses, and covers 2,568 Chinese characters. These samples were gathered across a diverse range of indoor and outdoor environments, capturing multiple viewing angles, varying light intensities, and different camera motions. Due to the absence of benchmark algorithms for comparison in this new task, we retrained and evaluated multiple state-of-the-art SLT algorithms, and believe that this benchmark can effectively support subsequent related research. Additionally, we propose a novel RGB-Event sign language translation framework (i.e., M$^2$-SLT) that incorporates fine-grained micro-sign and coarse-grained macro-sign retrieval, achieving state-of-the-art results on the proposed dataset. Both the source code and dataset will be released on https://github.com/Event-AHU/OpenESL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。