针对下游只用一种模态的问题，让预训练更偏向该模态，提升性能。
Finetune-Informed Pretraining Boosts Downstream Performance
- 预训练时对目标模态提高掩码难度、增强损失权重、增加解码器容量。
- 在无线信号星座图任务上，下游微调性能持续提升，无需额外数据或算力。
- 方法简单通用，适用于各类多模态掩码建模，不改编码器也不需新标注。
多模态预训练能构建通用表示，但在实际部署中，下游微调常仅使用单一模态。标准预训练对所有模态一视同仁，导致实际关键模态的表示未能充分优化。本文提出面向微调的预训练（FIP），一种模型无关的方法，通过在预训练阶段增强目标模态的掩码难度、损失权重和解码器容量，使其表示更适配后续微调需求。该方法不修改共享编码器，也无需额外监督。在无线信号星座图的掩码建模任务上，FIP显著提升下游微调性能，且无需额外数据或计算开销。FIP实现简单、架构兼容，可广泛应用于各类多模态掩码建模流程。
原文摘要 · Abstract (English)
Multimodal pretraining is effective for building general-purpose representations, but in many practical deployments, only one modality is heavily used during downstream fine-tuning. Standard pretraining strategies treat all modalities uniformly, which can lead to under-optimized representations for the modality that actually matters. We propose Finetune-Informed Pretraining (FIP), a model-agnostic method that biases representation learning toward a designated target modality needed at fine-tuning time. FIP combines higher masking difficulty, stronger loss weighting, and increased decoder capacity for the target modality, without modifying the shared encoder or requiring additional supervision. When applied to masked modeling on constellation diagrams for wireless signals, FIP consistently improves downstream fine-tuned performance with no extra data or compute. FIP is simple to implement, architecture-compatible, and broadly applicable across multimodal masked modeling pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。