针对微调模型的测试数据选择难题,提出高效精准的测试集筛选方法。
MetaSel: A Test Selection Approach for Fine-tuned DNN Models
- 利用微调前后模型行为差异识别易错样本,指导测试集选择
- 在标签预算极低时仍保持28.46%~56.18%的覆盖率提升
- 适合资源受限场景下微调模型的质量评估与验证
深度神经网络在部署时面临协变量偏移问题,即开发与部署阶段的数据分布不一致。微调可使预训练模型适应新环境,但受限于标注预算,对微调后模型的测试仍是关键挑战。本文提出MetaSel,一种专为应对协变量偏移而微调的DNN模型设计的测试选择方法。该方法基于微调前后模型共享相关数据分布、多数输入表现相似,但在决策边界被微调改变的子空间中行为差异显著的假设,通过比较两模型的行为差异来估计未标记测试样本的误分类概率,从而实现更有效的测试集选择。在涵盖弱、中、强分布偏移的68个微调模型上,与11种先进方法对比的实证评估表明,MetaSel在测试相对覆盖率(TRC)上持续显著优于基线,尤其在严苛标注预算下表现突出,平均提升达28.46%至56.18%,且中位数高、波动小。结果验证了MetaSel在实际应用中的有效性、鲁棒性与成本效益。
原文摘要 · Abstract (English)
Deep Neural Networks (DNNs) face challenges during deployment due to covariate shift, i.e., data distribution shifts between development and deployment contexts. Fine-tuning adapts pre-trained models to new contexts requiring smaller labeled sets. However, testing fine-tuned models under constrained labeling budgets remains a critical challenge. This paper introduces MetaSel, a new approach tailored for DNN models that have been fine-tuned to address covariate shift, to select tests from unlabeled inputs. MetaSel assumes that fine-tuned and pre-trained models share related data distributions and exhibit similar behaviors for many inputs. However, their behaviors diverge within the input subspace where fine-tuning alters decision boundaries, making those inputs more prone to misclassification. Unlike general approaches that rely solely on the DNN model and its input set, MetaSel leverages information from both the fine-tuned and pre-trained models and their behavioral differences to estimate misclassification probability for unlabeled test inputs, enabling more effective test selection. Our extensive empirical evaluation, comparing MetaSel against 11 state-of-the-art approaches and involving 68 fine-tuned models across weak, medium, and strong distribution shifts, demonstrates that MetaSel consistently delivers significant improvements in Test Relative Coverage (TRC) over existing baselines, particularly under highly constrained labeling budgets. MetaSel shows average TRC improvements of 28.46% to 56.18% over the most frequent second-best baselines while maintaining a high TRC median and low variability. Our results confirm MetaSel's practicality, robustness, and cost-effectiveness for test selection in the context of fine-tuned models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。