让机器自动选对OOD泛化算法,省去试错成本。
OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable?
- 用数据集特征训练分类器,自动选择最优的OOD训练算法。
- 在合成、视觉和语言任务上均能选出高性能算法。
- 发现算法适用规律,为理解现有方法提供新视角。
分布外(OOD)泛化挑战巨大,因分布偏移形式多样。已有多种算法针对特定场景,但缺乏有效方法在不试错的前提下为特定数据集选择最佳训练算法。本文提出概念验证模型 OOD-Chameleon,将算法选择建模为多标签分类问题,基于包含多种偏移的数据集集合进行训练。该模型仅凭数据集特征即可在未见偏移和数据集上对算法进行排序,无需先训练模型,突破传统模型选择依赖训练的限制。大量实验表明,该方法在合成数据、视觉和语言任务中均能识别出高性能算法。进一步分析显示,模型学习到非平凡的决策规则,揭示了现有算法的适用边界。该研究为更高效利用和理解现有OOD泛化算法提供了新思路。
原文摘要 · Abstract (English)
Out-of-distribution (OOD) generalization is challenging because distribution shifts come in many forms. Numerous algorithms exist to address specific settings, but choosing the right training algorithm for the right dataset without trial and error is difficult. Indeed, real-world applications often involve multiple types and combinations of shifts that are hard to analyze theoretically. Method. This work explores the possibility of learning the selection of a training algorithm for OOD generalization. We propose a proof of concept (OOD-Chameleon) that formulates the selection as a multi-label classification over candidate algorithms, trained on a dataset of datasets representing a variety of shifts. We evaluate the ability of OOD-Chameleon to rank algorithms on unseen shifts and datasets based only on dataset characteristics, i.e., without training models first, unlike traditional model selection. Findings. Extensive experiments show that the learned selector identifies high-performing algorithms across synthetic, vision, and language tasks. Further inspection shows that it learns non-trivial decision rules, which provide new insights into the applicability of existing algorithms. Overall, this new approach opens the possibility of better exploiting and understanding the plethora of existing algorithms for OOD generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。