实现低延迟的实时手语翻译系统，支持多种终端设备。
Toward Real-Time Sentence-Level Sign Language Translation
- 用QLoRA微调模型，保持编码器冻结以节省资源。
- 测试集上达到BLEU 15.9、BLEURT 44.7，响应延迟降低27.7%。
- 部署在树莓派上，可适配手机、网页等多端实时使用。
现有手语理解系统多基于孤立手势，难以用于自然交流。本文聚焦于句级手语翻译（SLT）的实时部署，而非提出新架构。在计算与存储限制下，从How2Sign数据集中选取9,872个样本，对SHuBERT-ByT5翻译栈采用QLoRA进行微调，同时保持SHuBERT冻结。模型在验证集上获得BLEU 16.7，测试集上达BLEU 15.9和BLEURT 44.7。核心贡献是一个面向硬件的流式系统：以树莓派4B为客户端，完成摄像头采集、本地文本显示与语音输出；感知与翻译任务由CPU/GPU后端处理。采集协议客户端无关，同一后端可服务浏览器、手机或笔记本。通过分块输入、有限队列、并行感知、时间重排序及句子边界状态机，平均延迟从1.873秒降至1.354秒（降幅27.71%），P95延迟从2.919秒降至2.130秒（降幅27.03%）。
原文摘要 · Abstract (English)
Most sign language understanding systems operate at the level of isolated signs, limiting their usefulness in natural communication. We study sentence-level sign language translation (SLT) with the primary goal of real-time deployment rather than proposing a new translation architecture. We fine-tune a SHuBERT-ByT5 translation stack on a uniformly sampled 9,872-example subset of How2Sign, selected because of compute and storage constraints, using QLoRA while keeping SHuBERT frozen. The model obtains a validation BLEU of 16.7 and, on the test split, BLEU 15.9 and BLEURT 44.7. The main contribution is a hardware-aware streaming system: a Raspberry Pi 4B reference client provides camera capture, local text display, and speech output, while compute-intensive perception and translation run on a CPU/GPU backend. The capture protocol remains client-agnostic, so the same backend can serve a browser, phone, or laptop. Chunked ingestion, bounded queues, parallelized perception, temporal reordering, and a sentence-boundary state machine reduce mean post-finalization response latency from 1.873 to 1.354 seconds (27.71%) and P95 latency from 2.919 to 2.130 seconds (27.03%) over the complete 9,872-example working subset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。