自适应筛选高质量遥感数据,用更少数据实现更好模型性能。
A Novel Adaptive Fine-Tuning Algorithm for Multimodal Models: Self-Optimizing Classification and Selection of High-Quality Datasets in Remote Sensing
- 先聚类再选数据:基于语义相似性分组,用向量差异选泛化强样本。
- 只用1/3数据训练,性能仅降1%,训练时间减少68.2%。
- 适合资源有限但追求高效高质遥感多模态模型的团队使用。
我们提出一种用于多模态大模型的自适应微调算法。该算法包含两阶段裁剪:首先将海量数据投影至语义向量空间,利用MiniBatchKMeans自动聚类,确保每簇内数据具有高语义相似性;其次在各簇中计算原始数据与扰动数据在多模态大模型向量空间中的平移差异,该差异作为数据泛化能力度量指标,据此选择具备高泛化潜力的数据用于训练。我们将该算法应用于InternLM-XComposer2-VL-7B模型,在两个3090 GPU上使用三分之一的GeoChat多模态遥感数据集进行训练。实验结果表明,本方法优于当前最优基线。基于实验验证,仅使用三分之一优化后数据训练的模型,在各类遥感评估指标上性能仅比全量数据训练模型下降1%。该方法显著保留了通用能力,同时将训练时间缩短68.2%。此外,模型在UCMerced和AID数据集上分别取得89.86和77.19的得分,较GeoChat数据集分别提升5.43和5.16点,而在LRBEN数据集上平均仅下降0.91点。
原文摘要 · Abstract (English)
We propose an adaptive fine-tuning algorithm for multimodal large models. The core steps of this algorithm involve two stages of truncation. First, the vast amount of data is projected into a semantic vector space, and the MiniBatchKMeans algorithm is used for automated clustering. This classification ensures that the data within each cluster exhibit high semantic similarity. Next, we process the data in each cluster, calculating the translational difference between the original and perturbed data in the multimodal large model's vector space. This difference serves as a generalization metric for the data. Based on this metric, we select the data with high generalization potential for training. We applied this algorithm to train the InternLM-XComposer2-VL-7B model on two 3090 GPUs using one-third of the GeoChat multimodal remote sensing dataset. The results demonstrate that our algorithm outperforms the state-of-the-art baselines. various baselines. The model trained on our optimally chosen one-third dataset, based on experimental validation, exhibited only 1% reduction in performance across various remote sensing metrics compared to the model trained on the full dataset. This approach significantly preserved general-purpose capabilities while reducing training time by 68.2%. Furthermore, the model achieved scores of 89.86 and 77.19 on the UCMerced and AID evaluation datasets, respectively, surpassing the GeoChat dataset by 5.43 and 5.16 points. It only showed a 0.91-point average decrease on the LRBEN evaluation dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。