用隐含思维链提升手语翻译的逻辑连贯性
Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation
- 引入潜空间思维序列作为视频与文本间的中间表示
- 计划-回溯解码使生成更连贯且忠实于视频内容
- 构建新数据集，强化上下文依赖与真实语义
许多手语翻译系统默认手势片段直接对应口语词汇，但实际中使用者常通过上下文、空间和动作即时构建意义。本文重新审视手语翻译，认为其本质是跨模态推理任务，而非简单的视频到文本转换。为此提出一种基于推理的框架，采用有序的潜空间思维序列作为视频与输出文本之间的显式中间层，逐步提取并组织语义。同时采用‘先规划后对齐’的解码策略：模型先决定要表达的内容，再回看视频寻找依据。该设计提升了生成结果的连贯性与忠实度。此外，构建并发布了一个大规模无词签手语翻译数据集，具有更强的上下文依赖和更真实的语义。在多个基准测试上，本方法显著优于现有无词签方法。代码与数据已公开于 https://github.com/fletcherjiang/SignThought。
原文摘要 · Abstract (English)
Many SLT systems quietly assume that brief chunks of signing map directly to spoken-language words. That assumption breaks down because signers often create meaning on the fly using context, space, and movement. We revisit SLT and argue that it is mainly a cross-modal reasoning task, not just a straightforward video-to-text conversion. We thus introduce a reasoning-driven SLT framework that uses an ordered sequence of latent thoughts as an explicit middle layer between the video and the generated text. These latent thoughts gradually extract and organize meaning over time. On top of this, we use a plan-then-ground decoding method: the model first decides what it wants to say, and then looks back at the video to find the evidence. This separation improves coherence and faithfulness. We also built and released a new large-scale gloss-free SLT dataset with stronger context dependencies and more realistic meanings. Experiments across several benchmarks show consistent gains over existing gloss-free methods. Our code and data are available at https://github.com/fletcherjiang/SignThought.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。