分模块识别手语动作、指拼和口型,提升翻译准确率。
SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation
- 分设专家网络分别处理手语、指拼和口型信息
- 融合后在三个数据集上达到新最佳性能
- 适合需要高精度手语翻译的科研与应用
尽管无词符手语翻译取得进展,传统单模态端到端方法在高速指拼识别和面部非手动线索整合方面仍存在明显不足。近期基于大语言模型的方法虽规避了该问题,但迫使单一网络同时学习多个任务,导致姓名、地名和技术术语等关键信息翻译效果差。我们提出 SignBind-LLM,一个模块化框架,分别采用专用预测器处理连续手语、指拼和唇读。各模态先解码为标记序列,再通过轻量级 Transformer 融合时序错位信息,最后输入大语言模型生成完整句子。该方法在 How2Sign、ChicagoFSWildPlus 和 BOBSL 数据集上分别取得 22.1 的 BLEU-4 得分、73.2% 的字母准确率和 6.8 的 BLEU-4 得分,创下新纪录。结果验证核心假设:先分离识别再融合,是实现高保真手语翻译的有效路径。
原文摘要 · Abstract (English)
Despite progress in gloss-free Sign Language Translation (SLT), traditional single modality end-to-end approaches consistently fail on two critical components of natural signing: the precise recognition of high-speed fingerspelling and the integration of asynchronous non-manual cues from the face. Recent progress in SLT with Large Language Models has side stepped this challenge, forcing a single network to learn these simultaneously resulting in poor performance when tasked with translating crucial information such as names, places, and technical terms. We introduce SignBind-LLM, a modular framework designed to overcome these limitations. Our approach employs separate, specialized predictors for continuous signing, fingerspelling, and lipreading. Each expert network first decodes its specific modality into a sequence of tokens. These parallel streams are then fused by a lightweight transformer that resolves temporal misalignments before passing the combined representation to a Large Language Model (LLM) for final sentence generation. Our method establishes a new state-of-the-art on the How2Sign, ChicagoFSWildPlus, and BOBSL datasets with a BLEU-4 score of 22.1, 73.2% letter accuracy and BLEU-4 score of 6.8 respectively. These results validate our core hypothesis: isolating and solving distinct recognition tasks before fusion provides a more powerful and effective pathway to robust, high-fidelity sign language translation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。