用错误分析指导训练，零样本高效适配多模态模型
Error-driven Data-efficient Large Multimodal Model Tuning
- 通过教师模型分析学生模型的推理错误，定位能力缺口
- 仅用通用数据集检索样本，实现7.01%平均性能提升
- 适合无标注数据的新任务快速部署，尤其适合资源受限场景
大型多模态模型（LMMs）在多个学术基准上表现优异，但下游任务仍需微调。然而，特定任务的训练样本通常难以获取，且收集成本高、耗时长。为此，我们提出一种基于错误驱动的数据高效微调框架，无需任何任务特定训练样本即可高效适配通用LMM以应对新任务。该方法首先让一个通用的LMM（作为学生模型）在目标任务的小规模验证集上进行评估，随后由一个更强的模型（作为教师模型）识别学生模型推理过程中的错误步骤，并分析其在完成目标任务上的能力差距。基于这些差距，从现有的任务无关数据集中检索针对性训练样本，用于微调学生模型，使其更贴合目标任务。我们在三种不同训练数据规模下对七个任务进行了广泛实验，结果表明，该训练范式显著且高效地提升了LMM在下游任务上的性能，平均性能提升达7.01%。
原文摘要 · Abstract (English)
Large Multimodal Models (LMMs) have demonstrated impressive performance across numerous academic benchmarks. However, fine-tuning still remains essential to achieve satisfactory performance on downstream tasks, while the task-specific tuning samples are usually not readily available or expensive and time-consuming to obtain. To address this, we propose an error-driven data-efficient tuning framework that aims to efficiently adapt generic LMMs to newly emerging tasks without requiring any task-specific training samples. In our approach, a generic LMM, acting as a student model, is first evaluated on a small validation set of the target task, and then a more powerful model, acting as a teacher model, identifies the erroneous steps within the student model's reasoning steps and analyzes its capability gaps from fully addressing the target task. Based on these gaps, targeted training samples are further retrieved from existing task-agnostic datasets to tune the student model and tailor it to the target task. We perform extensive experiments across three different training data scales and seven tasks, demonstrating that our training paradigm significantly and efficiently improves LMM's performance on downstream tasks, achieving an average performance boost of 7.01%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。