用语音生成带情绪的尼泊尔手语虚拟人,解决低资源语言难题
Low Resource Multimodal Translation of Nepali Spoken Words into Emotion-Conditioned Sign Language Avatars

- 共享声学编码器同时做语音识别和情绪分类
- 在600条语音上实现81.1%语音识别准确率
- 模型仅2210万参数,适合边缘设备部署
手语通信系统中融入情感表达仍属空白,尤其针对低资源语言。本研究提出NEST-V1(尼泊尔情感与语音变换器-版本1),一个概念验证型多模态框架,首次展示从口语输入生成带情绪的尼泊尔手语虚拟人的可行性。作为初步探索,我们选取四个常见尼泊尔词汇('谢谢'、'你好'、'房子'、'我')在三种情绪状态(快乐、中性、悲伤)下验证核心技术路径。系统采用轻量级架构,共享声学编码器同步执行自动语音识别与情绪分类,在50位说话者共600个标注音频样本上取得81.1%的语音识别准确率和79.21%的情绪识别准确率。相比独立模型架构,参数效率提升37%,总参数量仅22.1M,适合边缘部署。本工作为低资源环境下情感感知手语翻译奠定技术基础,并提供可扩展框架,未来可支持更大词库与更丰富情感表达。初步结果表明,实时、具情感表达的手语通信系统对听障群体具有可行性,后续开发路径清晰明确。
原文摘要 · Abstract (English)
Sign language communication systems, that integrate emotional expression remain underexplored, particularly for low-resource languages. This pilot study presents NEST-V1 (Nepali Emotion and Speech Transformer - Version 1), a proof-of-concept multimodal framework that demonstrates the feasibility of generating emotion-conditioned Nepali Sign Language avatars from spoken input. As a preliminary investigation, we focus on four common Nepali words ("thank you", "hello", "house", "me") across three emotional states (happy, neutral, sad) to validate our core technical approach. Our lightweight architecture employs a shared acoustic encoder for simultaneous Automatic Speech Recognition and emotion classification, achieving 81.1% ASR accuracy and 79.21% emotion recognition accuracy on a dataset of 600 labeled audio samples from 50 speakers. The system demonstrates 37% parameter efficiency compared to separate model architectures while maintaining a lightweight footprint with only 22.1M parameters suitable for edge deployment. This pilot work establishes the technical foundation for emotion-aware sign language translation in low-resource settings and provides a scalable framework for future expansion to larger vocabularies and more diverse emotional expressions. Our preliminary results indicate the viability of real-time, emotionally expressive sign language communication systems for the hearing-impaired community, with clear pathways for enhancement in subsequent development phases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。