用更少数据更快收敛,针对多模态指令微调优化训练样本。
Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning
- 基于目标驱动,为每样本计算六种特征,构建最优训练子集。
- 仅需35.4k~34.7k样本即达基线性能,准确率提升1.38~3.08个百分点。
- 适合追求高效训练的多模态模型开发者,尤其关注视频理解任务。
多模态指令微调常因计算效率低下,训练预算被分散在大量混合图像-视频数据中,而其效用差异显著。本文提出目标驱动的数据优化(GDO)框架,为每个候选样本计算六种描述符,并针对不同目标构建优化的1×训练子集。在8块H20 GPU上固定一轮Qwen3-VL-8B-Instruct训练与评估条件下,相比固定的512k样本基线(Uni-10x),GDO在MVBench、VideoMME、MLVU和LVBench上分别仅需35.4k、26.6k、27.3k、34.7k样本即可达到基线性能,准确率分别提升+1.38、+1.67、+3.08、+0.84个百分点。提升在MVBench和MLVU最显著,而LVBench改善较小,与其超长视频设置及训练数据以短视频/图像为主存在不匹配有关。在MinLoss、Diverse、Temp和Temp+策略中,更强的时间建模能力带来更优的长视频理解表现。整体上,GDO提供了一种在固定训练协议下实现更快收敛与更少样本消耗的目标驱动数据优化方案。代码已开源:https://github.com/rujiewu/GDO。
原文摘要 · Abstract (English)
Multimodal instruction tuning is often compute-inefficient because training budgets are spread across large mixed image-video pools whose utility is highly uneven. We present Goal-Driven Data Optimization (GDO), a framework that computes six sample descriptors for each candidate and constructs optimized 1$\times$ training subsets for different goals. Under a fixed one-epoch Qwen3-VL-8B-Instruct training and evaluation recipe on 8 H20 GPUs, GDO uses far fewer training samples than the Uni-10x baseline while converging faster and achieving higher accuracy. Relative to the fixed 512k-sample Uni-10x baseline, GDO reaches the Uni-10x reference after 35.4k samples on MVBench, 26.6k on VideoMME, 27.3k on MLVU, and 34.7k on LVBench, while improving Accuracy by +1.38, +1.67, +3.08, and +0.84 percentage points, respectively. The gains are largest on MVBench and MLVU, while LVBench improves more modestly, consistent with its ultra-long-video setting and the mismatch between that benchmark and the short-video/image-dominant training pool. Across MinLoss, Diverse, Temp, and Temp+, stronger temporal emphasis yields steadily better long-video understanding behavior. Overall, GDO provides a goal-driven data optimization framework that enables faster convergence with fewer training samples under a fixed training protocol. Code is available at https://github.com/rujiewu/GDO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。