提出可定位并解释大模型性能下降根源的分层诊断框架
"Who experiences large model decay and why?" A Hierarchical Framework for Diagnosing Heterogeneous Performance Drift
- 构建分层推理框架，先定位受损子群，再分析具体变量变化
- 实测发现部分子群性能下降超30%，且与特定变量漂移相关
- 适合需要精准修复模型偏差的研究者和工程师
机器学习模型在新场景部署时常出现性能下降，且衰减程度不均：某些子群性能大幅下滑而其他子群则不受影响。理解性能差异的分布位置及成因，对制定针对性修正策略至关重要。现有方法要么仅解释平均性能变化，要么识别受影响子群但无法说明原因。为此，我们提出子群扫描分层推断框架SHIFT：首先判断是否存在因协变量/结果变量漂移导致性能显著下降的子群（何处？），若有，则进一步探究是否可通过更细粒度的变量（子集）特异性漂移来解释（如何？）。在真实场景实验中，SHIFT成功识别出可解释的受损子群，并提出有效缓解性能下降的精准干预措施。
原文摘要 · Abstract (English)
Machine learning (ML) models frequently experience performance degradation when deployed in new contexts. Such degradation is rarely uniform: some subgroups may suffer large performance decay while others may not. Understanding where and how large differences in performance arise is critical for designing targeted corrective actions that mitigate decay for the most affected subgroups while minimizing any unintended effects. Current approaches do not provide such detailed insight, as they either (i) explain how average performance shifts arise or (ii) identify adversely affected subgroups without insight into how this occurred. To this end, we introduce a Subgroup-scanning Hierarchical Inference Framework for performance drifT (SHIFT). SHIFT first asks "Is there any subgroup with unacceptably large performance decay due to covariate/outcome shifts?" (Where?) and, if so, dives deeper to ask "Can we explain this using more detailed variable(subset)-specific shifts?" (How?). In real-world experiments, we find that SHIFT identifies interpretable subgroups affected by performance decay, and suggests targeted actions that effectively mitigate the decay.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。