让模型在任意分布下都表现稳定，只需均匀训练数据。
Generalizing to any diverse distribution: uniformity, gentle finetuning and rebalancing
- 用域内均匀分布训练，可最大化泛化能力。
- 实验证明，均匀训练能有效应对分布外挑战。
- 适合追求强鲁棒性的模型开发者参考。
随着训练数据集规模扩大，我们希望模型能在任何多样化的测试分布上良好泛化，即使该分布与训练数据差异显著。现有方法如领域自适应、领域泛化和鲁棒优化，通常依赖训练与测试分布间关系的假设。本文采取更保守视角，考虑域内所有足够多样的测试分布下的最坏情况误差。首个发现是：在该域的均匀分布上训练是最优策略。当均匀样本不可得时，我们研究通过微调和重加权来缓解非均匀性。理论为先前关于熵和重加权在分布外泛化及基础模型训练中的作用提供了数学依据，并在多个涉及分布外偏移的任务中提供了新实证证据，验证了该视角的广泛适用性。
原文摘要 · Abstract (English)
As training datasets grow larger, we aspire to develop models that generalize well to any diverse test distribution, even if the latter deviates significantly from the training data. Various approaches like domain adaptation, domain generalization, and robust optimization attempt to address the out-of-distribution challenge by posing assumptions about the relation between training and test distribution. Differently, we adopt a more conservative perspective by accounting for the worst-case error across all sufficiently diverse test distributions within a known domain. Our first finding is that training on a uniform distribution over this domain is optimal. We also interrogate practical remedies when uniform samples are unavailable by considering methods for mitigating non-uniformity through finetuning and rebalancing. Our theory provides a mathematical grounding for previous observations on the role of entropy and rebalancing for o.o.d. generalization and foundation model training. We also provide new empirical evidence across tasks involving o.o.d. shifts which illustrate the broad applicability of our perspective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。