arXiv:2604.08192cs.LGcs.CV2026-04

通过分析模型内部结构,提升对视觉模型泛化能力的评估可靠性。

Inside-Out: Measuring Generalization in Vision Transformers Through Inner Workings

  • 从模型内部表示间的因果关系中提取电路特征作为评估依据。
  • 提出两个新指标,分别用于部署前选型和部署后监控,相关性提升超30%。
  • 适合需要无标签评估泛化性能的研究者与工程师使用。

可靠的泛化性能评估指标对机器学习模型至关重要,尤其在目标数据标注稀缺的高风险场景下。本文针对两个实际问题:(1)部署前如何为未标注的目标数据选择最优模型?(2)部署后如何监测模型在分布偏移下的性能?核心需求是可靠且无需标签的代理指标。现有方法如模型置信度或线性准确率常不可靠,因仅关注输出而忽略内部机制。为此,本文提出新视角:利用模型内部运作机制——即内部电路——作为泛化性能的预测指标。基于电路发现技术,我们提取内部表示间的因果交互形成电路,并据此构建两个针对性指标:(1)部署前采用依赖深度偏差(Dependency Depth Bias)衡量模型在目标数据上的泛化能力;(2)部署后采用电路漂移评分(Circuit Shift Score)预测模型在不同分布偏移下的表现。在多个任务上,两项指标与真实泛化性能的相关性显著提升,平均优于现有代理指标13.4%和34.1%。代码已开源:https://github.com/deep-real/GenCircuit。

原文摘要 · Abstract (English)

Reliable generalization metrics are fundamental to the evaluation of machine learning models. Especially in high-stakes applications where labeled target data are scarce, evaluation of models' generalization performance under distribution shift is a pressing need. We focus on two practical scenarios: (1) Before deployment, how to select the best model for unlabeled target data? (2) After deployment, how to monitor model performance under distribution shift? The central need in both cases is a reliable and label-free proxy metric. Yet existing proxy metrics, such as model confidence or accuracy-on-the-line, are often unreliable as they only assess model output while ignoring the internal mechanisms that produce them. We address this limitation by introducing a new perspective: using the inner workings of a model, i.e., circuits, as a predictive metric of generalization performance. Leveraging circuit discovery, we extract the causal interactions between internal representations as a circuit, from which we derive two metrics tailored to the two practical scenarios. (1) Before deployment, we introduce Dependency Depth Bias, which measures different models' generalization capability on target data. (2) After deployment, we propose Circuit Shift Score, which predicts a model's generalization under different distribution shifts. Across various tasks, both metrics demonstrate significantly improved correlation with generalization performance, outperforming existing proxies by an average of 13.4\% and 34.1\%, respectively. Our code is available at https://github.com/deep-real/GenCircuit.

视觉模型泛化评估内部机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。