构建首个大规模日语多模态数据集,提升视觉语言模型在日文任务表现。
Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models
- 通过图像、文本对和文档生成920万条日语多模态问答数据
- 用Jagle训练的22亿模型在10项日文任务平均分超InternVL3.5-2B
- 适配多种模型架构,助力日语与英语双语性能提升
构建能跨任务泛化的视觉语言模型(VLMs)需要大规模、内容多样化的训练数据。在英语中,这类数据集通常通过聚合和整理现有视觉问答(VQA)资源构建,但该方法难以扩展至其他语言,因日语等语言的VQA数据集规模小、覆盖范围有限。本文提出Jagle,目前最大的日语多模态后训练数据集,包含约920万条跨多样化任务的实例。我们未依赖现有VQA数据,而是收集图像、图文对和PDF文档等异构源数据,采用基于VLM的问答生成、翻译和文本渲染等多种策略自动构建VQA对。实验表明,使用Jagle训练的2.2B模型在十项日文评估任务上的平均得分超过InternVL3.5-2B,并接近Qwen3-VL-2B-Instruct(差5分以内)。将Jagle与FineVision结合可提升英语性能,优于仅用FineVision训练。该优势在不同模型架构和规模上均成立:基于日语主干模型LLM-jp-4-8B-instruct构建的90亿参数模型LLM-jp-4-VL 9B beta,在日文表现上媲美Qwen3-VL-8B-Instruct。为促进复现与研究,我们开源了数据集、模型及代码。
原文摘要 · Abstract (English)
Developing vision-language models (VLMs) that generalize across diverse tasks requires large-scale training datasets with diverse content. In English, such datasets are typically constructed by aggregating and curating numerous existing visual question answering (VQA) resources. However, this strategy does not readily extend to other languages, where VQA datasets remain limited in both scale and domain coverage. In this work, we introduce Jagle, the largest Japanese multimodal post-training dataset to date, comprising approximately 9.2 million instances across diverse tasks. Rather than relying on existing VQA datasets, we collect heterogeneous source data, including images, image-text pairs, and PDF documents, and generate VQA pairs through multiple strategies such as VLM-based QA generation, translation, and text rendering. Experiments demonstrate that a 2.2B model trained with Jagle surpasses InternVL3.5-2B in average score across ten Japanese evaluation tasks and comes within five points of Qwen3-VL-2B-Instruct. Moreover, combining Jagle with FineVision improves English performance over training with FineVision alone. These benefits generalize across model backbones and scales: LLM-jp-4-VL 9B beta, a 9B-scale model built on the Japanese-centric LLM-jp-4-8B-instruct, achieves Japanese performance comparable to Qwen3-VL-8B-Instruct. To facilitate reproducibility and future research, we release the dataset, models, and code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。