提出Q-BridgeNet框架，实现多语言手语翻译的统一建模与跨语言冲突缓解。
Q-BridgeNet: A Quantization Network for Cross-Lingual Sign Language Translation
- 通过自适应分段与残差向量量化，构建共享语义基码本与语言特异残差码本。
- 在三个数据集上均达到当前最优性能，非本地对翻译也表现出强泛化能力。
- 适合需要多语言手语翻译系统的开发者与无障碍研究者使用。
多数手语翻译（SLT）方法聚焦于孤立的母语手语-口语对（如美国手语-英语）。将语言特定的SLT模型扩展至多语言翻译，可提升跨不同手语与口语社区交流的可及性。然而，现有方法仍难以在最小化跨语言冲突的同时，捕捉共享的跨语言语义并保留各手语的语言特异性。为此，我们提出Q-BridgeNet，一种统一的多语言SLT框架，联合缓解手语与口语两端的跨语言冲突。在手语端，Q-BridgeNet通过自适应分段和残差向量量化学习离散的Q-unit：共享基码本提供无语言差异的语义原语，而语言特异残差码本细化异构的手语语义。在口语端，微调多语言大模型以在Q-unit空间运行，利用跨语言先验实现统一的SLT模型。在PHOENIX14T、How2Sign和CSL-Daily上的实验表明，Q-BridgeNet有效缓解跨语言冲突，在本地手语-口语对上达到领先性能，并展现出对非本地对的强大泛化能力。源代码已公开：https://github.com/FengLiQ/Q-BridgeNet
原文摘要 · Abstract (English)
Most sign language translation (SLT) methods focus on isolated native sign-spoken pairs (e.g., American Sign Language - English). Extending language-specific SLT models to multilingual translation would improve accessibility by enabling communication across diverse sign and spoken language communities. However, existing multilingual SLT approaches still struggle to learn a unified model that minimizes cross-lingual conflicts while capturing shared cross-lingual semantics and preserving language-specific variations across different sign languages. Therefore, we propose Q-BridgeNet, a unified framework for multilingual SLT that jointly mitigates cross-lingual conflicts across both the sign language and spoken language sides. On the sign language side, Q-BridgeNet learns discrete Q-units via adaptive segmentation and residual vector quantization: a shared base codebook provides language-agnostic semantic primitives, while language-specific residual codebooks refine heterogeneous signing semantics. On the spoken language side, a multilingual LLM is fine-tuned to operate in the Q-unit space, leveraging cross-lingual priors to enable a unified SLT model. Experiments on PHOENIX14T, How2Sign, and CSL-Daily show that Q-BridgeNet effectively mitigates cross-lingual conflicts, achieving state-of-the-art performance on native sign-spoken pairs while also demonstrating strong generalization to non-native pairs. Our source code is publicly available at: https://github.com/FengLiQ/Q-BridgeNet
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。