提出BGR增广方法，有效区分模型是否被污染，提升视觉语言模型评估可信度。
Clean Evaluations on Contaminated Visual Language Models
- 用BGR通道切换实现视觉输入增广，防止数据污染影响评估结果。
- 在数千样本的新基准上验证，传统增广易被训练数据吸收，而BGR不易。
- 该方法难被恶意训练者利用，适合用于真实场景下的模型性能评测。
如何对大语言模型进行无污染评估已成为研究重点，但视觉语言模型（VLM）的清洁评估仍缺乏系统研究。本文通过视觉输入的数据增强方法，提出一种新思路，并构建包含数千个实例的视觉清洁评估基准。实验发现，传统视觉增强虽有效，但可能被用作训练数据的规避手段。为此，我们引入BGR通道切换作为增广策略，发现其能显著降低数据污染影响，且因其自身难以融入训练过程，恶意训练者无法利用。这使得BGR增广成为评估视觉语言模型性能时具有高可信度的工具。代码、数据及模型权重将在论文发表后公开。
原文摘要 · Abstract (English)
How to evaluate large language models (LLMs) cleanly has been established as an important research era to genuinely report the performance of possibly contaminated LLMs. Yet, how to cleanly evaluate the visual language models (VLMs) is an under-studied problem. We propose a novel approach to achieve such goals through data augmentation methods on the visual input information. We then craft a new visual clean evaluation benchmark with thousands of data instances. Through extensive experiments, we found that the traditional visual data augmentation methods are useful, but they are at risk of being used as a part of the training data as a workaround. We further propose using BGR augmentation to switch the colour channel of the visual information. We found that it is a simple yet effective method for reducing the effect of data contamination and fortunately, it is also harmful to be used as a data augmentation method during training. It means that it is hard to integrate such data augmentation into training by malicious trainers and it could be a promising technique to cleanly evaluate visual LLMs. Our code, data, and model weights will be released upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。