通过随机调整对话轮次深度，提升视觉语言模型在多轮对话中的表现。
StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning
- 用随机分组图像任务的轮次深度来训练模型，避免注意力衰减。
- 在单轮和多轮测试中均显著提升模型性能，准确率提高7.3%。
- 适合需要多轮交互能力的视觉语言模型训练与评估。
大型视觉语言模型（LVLMs）依赖视觉指令微调（VIT）来激发其多模态推理能力。然而我们发现：VIT常对同一图像打包多个语言任务进行多轮对话训练，而现有基准测试却在孤立的单轮场景下评估模型。这导致模型在多轮训练中易出现视觉注意力衰减和上下文过拟合，难以在不匹配的测试阶段发挥全部潜力。为此，我们提出学习时的随机轮次深度（StochasT），随机将同一图像的语言任务聚类为不同规模（轮次深度）的组块，同时保持其自然顺序。虽借鉴了残差网络中的随机深度和Dropout思想，但并非实际丢弃数据，以最大化训练数据利用率。此外，我们引入基于平衡拉丁方的、与基准无关的评估机制，以衡量模型在不同上下文依赖下的鲁棒性。大量实验表明，StochasT有效赋予LVLMs在单轮与多轮场景下均强大的均衡能力。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) rely extensively on Visual Instruction Tuning (VIT) to elicit their multimodal reasoning capabilities. However, we find a discrepancy: VIT often packs multiple language tasks about the same image for conversational, multi-turn training, whereas existing benchmarks evaluate LVLMs in isolated, single-turn scenarios. The models can suffer from visual attention decay and contextual overfitting during multi-turn training, making it hard for them to realize their full potential in the mismatched test phase. To close the gap, we propose learning with Stochastic Turn Depth (StochasT), which stochastically groups language tasks for the same image into clusters of varying sizes (turn depth) while preserving their organic order. Hence, while StochasT draws on Dropout and stochastic depth for ResNets, it does not actually drop anything to maximize the utility of the training data. Furthermore, we introduce a challenging, benchmark-agnostic evaluation mechanism based on the Balanced Latin Square to measure LVLMs' robustness under varying contextual dependencies. Extensive experiments demonstrate that StochasT effectively grants LVLMs strong, harmonized capabilities for both single-turn and multi-turn use cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。