arXiv:2501.11039cs.LG2025-01被引 21

用预测模型筛选难任务,让大模型更高效稳定地适应新场景。

Model Predictive Task Sampling for Efficient and Robust Adaptation

  • 基于生成模型预测任务适应风险,替代昂贵的试错评估。
  • 在模式识别和决策任务中显著提升对罕见/分布外任务的鲁棒性。
  • 适合机器人、大模型微调等评估成本高或风险大的场景。

基础模型通过预训练、元训练和微调实现了通用问题求解的快速适配。近期研究发现,在分布偏移下,优先采样挑战性任务可提升适应鲁棒性。然而,迭代中对任务难度排序通常需耗时的任务评估,实际中难以承受。本文提出模型预测任务采样(MPTS),将任务空间与适应风险分布相连接,为鲁棒主动任务采样提供理论基础。MPTS利用生成模型刻画周期优化过程,通过后验推断预测任务特定适应风险,其风险预测模型可降低任务适应性能评估开销,并严格逼近任务难度排序。该方法可无缝融入零样本、少样本及监督微调设置。我们在基础模型的模式识别与序列决策任务上进行广泛实验,结果表明MPTS显著提升对尾部风险或分布外(OOD)任务的适应鲁棒性,并优于当前最优方法,同时提高学习效率。代码已公开于https://github.com/thu-rllab/MPTS。

原文摘要 · Abstract (English)

Foundation models have revolutionized general-purpose problem-solving, offering rapid task adaptation through pretraining, meta-training, and finetuning. Recent crucial advances in these paradigms reveal the importance of challenging task prioritized sampling to enhance adaptation robustness under distribution shifts. However, ranking task difficulties over iteration as a preliminary step typically requires exhaustive task evaluation, which is practically unaffordable in computation and data-annotation. This study provides a novel perspective to illuminate the possibility of leveraging the dual importance of adaptation robustness and learning efficiency, particularly in scenarios where task evaluation is risky or costly, such as iterative agent-environment interactions for robotic policy evaluation or computationally intensive inference steps for finetuning foundation models. Firstly, we introduce Model Predictive Task Sampling (MPTS), a framework that bridges the task space and adaptation risk distributions, providing a theoretical foundation for robust active task sampling. MPTS employs a generative model to characterize the episodic optimization process and predicts task-specific adaptation risk via posterior inference. The resulting risk predictive model amortizes the costly evaluation of task adaptation performance and provably approximates task difficulty rankings. MPTS seamlessly integrates into zero-shot, few-shot, and supervised finetuning settings. Empirically, we conduct extensive experiments in pattern recognition using foundation models and sequential decision-making. Our results demonstrate that MPTS significantly enhances adaptation robustness for tail risk or out-of-distribution (OOD) tasks and improves learning efficiency compared to state-of-the-art (SoTA) methods. The code is available at the project site https://github.com/thu-rllab/MPTS.

大模型微调主动学习鲁棒性任务采样

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。