用下游任务反馈指导预训练,让模型学会预测该学什么。
Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining
- 设计轻量任务生成器,根据下游表现预测最佳自监督目标。
- 仅用1024个样本反馈,使Qwen模型在GSM8K上提升7.4分。
- 适用于希望优化特定任务性能的AI研发者,不破坏通用能力。
持续预训练通常采用固定自监督任务,但通过下游表现选择,形成粗粒度反馈循环:研究者需评估检查点、调整数据混合或目标后重启训练,而每次更新却无法感知目标能力变化。本文提出V-pretraining,将仅用自监督损失训练的模型与轻量级任务设计者解耦。给定当前模型和未标注批次,任务设计者通过预测所构造目标在自监督更新后对下游损失的一阶下降值,来评估候选构造方案,并最大化该值;随后模型使用生成的目标或视图进行更新,且下游标签从不参与参数更新。在语言和视觉模态中均实现目标能力提升而不降低泛化性。在相同耗时条件下,仅用1,024个GSM8K样本作为反馈,使Qwen模型在GSM8K Pass@1上提升7.4分,包括Qwen2.5-0.5B单次运行+7.4分增益;在视觉方面,提升DINOv3在ADE20K语义分割和NYUv2深度估计的迁移性能,同时保持ImageNet线性准确率,表明反馈引导的任务构建可在不破坏通用表征的前提下提升目标任务能力。
原文摘要 · Abstract (English)
Continued pretraining is optimized with fixed self-supervised tasks but selected by downstream performance, creating a coarse feedback loop in which practitioners evaluate checkpoints, change data mixtures or objectives, and restart runs, while individual updates remain blind to target capabilities. We ask whether a small set of verifiable downstream examples can provide step-level feedback without directly supervising the learner. We introduce V-pretraining, which decouples a learner trained only with a self-supervised loss from a lightweight task designer that constructs targets or views for unlabeled batches. Given the current learner and batch, V-pretraining scores a candidate construction by predicting the first-order reduction in downstream loss after the induced self-supervised update. The designer maximizes this value; the learner then applies the update with targets or views detached, so downstream labels never update learner parameters. We instantiate V-pretraining as adaptive top-K soft targets for language modeling and learned views or masks for self-supervised vision. Across both modalities, V-pretraining improves target capabilities without degrading generalization. Under wall-clock-matched continued pretraining, it improves GSM8K Pass@1 for Qwen models using 1,024 GSM8K examples only as feedback, including a +7.4 point single-run gain for Qwen2.5-0.5B. In vision, it improves DINOv3 transfer to ADE20K semantic segmentation and NYUv2 depth estimation while preserving ImageNet linear accuracy, suggesting that feedback-guided task construction can improve target capabilities without collapsing general-purpose representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。