ViDove让翻译模型能看图识境，提升字幕与文本翻译质量。
ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoning
- 融合视觉与上下文信息，模拟人类译者工作流程。
- 在字幕生成中BLEU提升28%，SubER降低15%。
- 适合视频翻译、多模态任务研究者使用。
基于大语言模型的翻译代理已实现接近人类水平的翻译效果，能高效处理长且复杂的语境。然而，它们通常仅限于纯文本输入。本文提出ViDove，一种面向多模态输入的翻译代理系统。受人类译者工作流程启发，ViDove利用视觉信息和上下文背景增强翻译过程。此外，系统集成多模态记忆系统与长短期记忆模块，并融入领域知识，使代理在真实场景中表现更准确、自适应。实验显示，ViDove在字幕生成与通用翻译任务中显著优于现有最先进基线，BLEU分数提升28%，SubER降低15%。同时，我们构建了DoveBench基准，包含17小时高质量人工标注数据，用于长篇自动视频字幕与翻译评估。代码已开源：https://github.com/pigeonai-org/ViDove。
原文摘要 · Abstract (English)
LLM-based translation agents have achieved highly human-like translation results and are capable of handling longer and more complex contexts with greater efficiency. However, they are typically limited to text-only inputs. In this paper, we introduce ViDove, a translation agent system designed for multimodal input. Inspired by the workflow of human translators, ViDove leverages visual and contextual background information to enhance the translation process. Additionally, we integrate a multimodal memory system and long-short term memory modules enriched with domain-specific knowledge, enabling the agent to perform more accurately and adaptively in real-world scenarios. As a result, ViDove achieves significantly higher translation quality in both subtitle generation and general translation tasks, with a 28% improvement in BLEU scores and a 15% improvement in SubER compared to previous state-of-the-art baselines. Moreover, we introduce DoveBench, a new benchmark for long-form automatic video subtitling and translation, featuring 17 hours of high-quality, human-annotated data. Our code is available here: https://github.com/pigeonai-org/ViDove
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。