融合语音与影像信息，提升实时核磁发音结构分割精度
VocSegMRI: Multimodal Learning for Precise Vocal Tract Segmentation in Real-time MRI
- 引入语音与音位信息，通过交叉注意力融合多模态特征
- 在USC-75数据集上达到91.3%的分割准确率，优于基线方法
- 适用于需要高精度发音动态分析的研究，如语音病理学
实时核磁共振（rtMRI）中精确分割发音器官仍具挑战，现有方法主要依赖视觉线索，忽视同步语音信号提供的互补信息。本文提出VocSegMRI，一种融合视频、音频和音位输入的多模态框架，采用交叉注意力机制与对比学习目标，增强跨模态对齐与分割精度。在USC-75数据集上评估，并通过零样本迁移在USC-TIMIT上验证，结果表明其性能超越单模态及多模态基线模型；消融实验确认各模块贡献。该方法显著提升发音结构分割准确性。
原文摘要 · Abstract (English)
Accurate segmentation of articulatory structures in real-time MRI (rtMRI) remains challenging, as existing methods rely primarily on visual cues and overlook complementary information from synchronized speech signals. We propose VocSegMRI, a multimodal framework integrating video, audio, and phonological inputs via cross-attention fusion and a contrastive learning objective that improves cross-modal alignment and segmentation precision. Evaluated on USC-75 and further validated via zero-shot transfer on USC-TIMIT, VocSegMRI outperforms unimodal and multimodal baselines, with ablations confirming the contribution of each component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。