双螺旋结构提升音视频语音识别准确率，尤其在嘈杂环境更鲁棒
DoubleHelix: Structured Cross-Modal Fusion for Audio-Visual Speech Recognition with LLMs
- 用多轮迭代交互替代单步融合，学习对齐约束实现结构化跨模态融合
- 在LRS3数据集上清洁音频下错误率降至0.68%，相对提升5.6%
- 适合做音视频语音识别、噪声环境鲁棒性研究的工程师和研究人员
音视频语音识别依赖于音频与视觉模态的有效融合，但现有方法将跨模态交互视为单步操作，缺乏结构化迭代优化。本文提出DoubleHelix框架，将融合重构为带有自适应降质感知增强的迭代跨模态交互过程。该框架包含三个组件：ReverseParallelHelix实现多轮结构化交互并引入学习对齐约束；QualitySensor用于学习降质感知门控信号；HelixReplication通过一致性引导实现条件特征增强。在LRS3数据集上的实验表明，DoubleHelix在相同主干网络设置下，清洁音频下的词错误率为0.68%，相比此前最优结果相对提升5.6%。全面消融实验证明各组件贡献显著，包括对非对称路径加权等设计选择的针对性分析。在背景噪声条件下表现更优，于SNR -5dB时达到11.6%词错误率。
原文摘要 · Abstract (English)
Audio-visual speech recognition (AVSR) relies on effective fusion of audio and visual modalities, yet existing approaches treat cross-modal interaction as a single-step operation without structured iterative refinement. We present DoubleHelix, a multimodal fusion framework that reformulates fusion as an iterative cross-modal interaction process with adaptive degradation-aware enhancement. The framework comprises three components including ReverseParallelHelix for multi-turn structured interaction with learned alignment constraints, QualitySensor for learning degradation-aware gating signals, and HelixReplication for consistency-guided conditional feature enhancement. Experiments on LRS3 demonstrate that DoubleHelix achieves 0.68% WER on clean audio, outperforming previous best results by 5.6% relative improvement under matched backbone settings. Comprehensive ablation studies validate each component contribution, including targeted analysis of design choices such as asymmetric pathway weighting. The framework shows improved robustness under evaluated babble-noise conditions, achieving 11.6% WER at SNR -5dB.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。