让大模型更懂图片里的文字,实现多语言网页图文翻译
VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image Translation

- 用双流注意力融合语言与视觉细节,生成统一特征
- 通过轻量适配器注入视觉信息,翻译准确率显著提升
- 适合做跨语言图文理解、电商与社交媒体内容处理
将嵌入网络图片中的文本进行翻译,对提升内容可访问性和跨语言信息检索至关重要,尤其在社交网络和电子商务领域。尽管大型视觉语言模型(LVLMs)已提升多模态理解能力,但将其应用于网页图像翻译仍面临视觉表征鸿沟:标准编码器常忽略细粒度视觉细节,难以识别多样化的字符形态。为此,我们提出VaaWIT,一个端到端的多语言网页图像翻译框架。该框架引入两项关键技术:(1) 双流注意力模块(DSAM),促进多语言语义特征与详细视觉表示间的双向交互,合成对文本变化鲁棒的统一特征;(2) 视觉感知适配器(VAA),一种参数高效微调策略,动态将融合后的视觉线索注入冻结的LLM主干。该设计使模型能有效对齐视觉上下文与语言推理,同时降低计算开销。在三个公开基准上的八个任务上,实验表明VaaWIT显著优于现有开源基线,并达到与专有模型相当的性能,验证了将细粒度视觉感知融入大模型在复杂网络内容分析中的有效性。
原文摘要 · Abstract (English)
Translating text embedded in Web images is crucial for improving content accessibility and cross-lingual information retrieval, particularly within social media and e-commerce domains. Although Large Vision-Language Models (LVLMs) have advanced multimodal understanding, applying them to Web image translation remains challenging due to the visual representation gap: standard encoders often prioritize high-level semantics over the fine-grained visual details required for recognizing diverse character morphologies. To address this challenge, we propose VaaWIT, an end-to-end framework that adapts Large Language Models for multilingual Web image translation. The framework introduces two key technical contributions: (1) a Dual-Stream Attention Module (DSAM), which facilitates bidirectional interaction between multilingual semantic features and detailed visual representations, thereby synthesizing unified features robust to textual variations; and (2) a Visual-Aware Adapter (VAA), a parameter-efficient fine-tuning strategy that dynamically injects these fused visual cues into the frozen LLM backbone. This design enables the model to align the visual context with linguistic reasoning effectively while minimizing computational costs. Extensive experiments on eight tasks on three public benchmarks demonstrate that VaaWIT significantly outperforms state-of-the-art (SOTA) open-source baselines and achieves competitive performance against proprietary models. These results validate the efficacy of integrating fine-grained visual perception into LLMs for complex Web content analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。