用生成图像提升翻译鲁棒性，无需真实配对图像
Dual-branch Prompting for Multimodal Machine Translation
- 双分支提示框架，结合真实与生成图像进行联合训练
- 在Multi30K上超越现有方法，有效抑制视觉噪声干扰
- 适合需要抗干扰能力的多模态翻译场景
多模态机器翻译通常通过引入对齐的视觉特征来增强纯文本翻译。尽管取得显著进展，当前主流方法在推理时仍依赖配对的图文输入，且对无关视觉噪声敏感，限制了其鲁棒性和实用性。为此，我们提出基于扩散模型的D2P-MMT框架，仅需源文本和由预训练扩散模型生成的重构图像，自然过滤冗余视觉细节，保留语义线索。训练阶段采用双分支提示策略，联合学习真实图像与重构图像的表示，促进跨模态交互。为弥合模态差异并减少训练-推理不一致，引入分布对齐损失，强制两分支输出分布一致。在Multi30K数据集上的大量实验表明，D2P-MMT性能优于现有最优方法。代码已公开于https://github.com/MentaY/DDP。
原文摘要 · Abstract (English)
Multimodal Machine Translation (MMT) typically enhances text-only translation by incorporating aligned visual features. Despite the remarkable progress, state-of-the-art MMT approaches often rely on paired image-text inputs at inference and are sensitive to irrelevant visual noise, which limits their robustness and practical applicability. To address these issues, we propose D2P-MMT, a diffusion-based dual-branch prompting framework for robust vision-guided translation. Specifically, D2P-MMT requires only the source text and a reconstructed image generated by a pre-trained diffusion model, which naturally filters out distracting visual details while preserving semantic cues. During training, the model jointly learns from both authentic and reconstructed images using a dual-branch prompting strategy, encouraging rich cross-modal interactions. To bridge the modality gap and mitigate training-inference discrepancies, we introduce a distributional alignment loss that enforces consistency between the output distributions of the two branches. Extensive experiments on the Multi30K dataset demonstrate that D2P-MMT achieves superior translation performance compared to existing state-of-the-art approaches. Our code is publicly available at https://github.com/MentaY/DDP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。