提出渐进式强化对齐方法,让视觉语言模型学新领域时不忘通用能力。
Reinforced Curriculum Pre-Alignment for Domain-Adaptive VLMs
- 分阶段施加输出约束,逐步引导模型适应新领域
- 在医学与几何等场景中显著提升性能,同时保持通用能力
- 适合需要兼顾专业性与泛化性的视觉语言模型部署
视觉语言模型(VLMs)具备强大的通用能力,但在医疗影像或几何推理等专业领域表现不佳。监督微调(SFT)虽可提升特定领域性能,但常引发灾难性遗忘,限制泛化能力。现有持续预训练因计算成本高且缺乏数据,难以应用于多数开源VLM。为此,本文提出强化课程预对齐(RCPA),一种新型后训练范式,引入课程感知的渐进调制机制。初期通过部分输出约束安全引入新领域概念;随着模型熟悉度提升,逐步转为全生成优化,精炼回答并对齐领域偏好。该方法在多个专业领域和通用基准上验证有效,实现了领域适应与通用能力平衡,为构建高性能、领域自适应的VLM提供了可行路径。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) demonstrate remarkable general-purpose capabilities but often fall short in specialized domains such as medical imaging or geometric problem-solving. Supervised Fine-Tuning (SFT) can enhance performance within a target domain, but it typically causes catastrophic forgetting, limiting its generalization. The central challenge, therefore, is to adapt VLMs to new domains while preserving their general-purpose capabilities. Continual pretraining is effective for expanding knowledge in Large Language Models (LLMs), but it is less feasible for VLMs due to prohibitive computational costs and the unavailability of pretraining data for most open-source models. This necessitates efficient post-training adaptation methods. Reinforcement learning (RL)-based approaches such as Group Relative Policy Optimization (GRPO) have shown promise in preserving general abilities, yet they often fail in domain adaptation scenarios where the model initially lacks sufficient domain knowledge, leading to optimization collapse. To bridge this gap, we propose Reinforced Curriculum Pre-Alignment (RCPA), a novel post-training paradigm that introduces a curriculum-aware progressive modulation mechanism. In the early phase, RCPA applies partial output constraints to safely expose the model to new domain concepts. As the model's domain familiarity increases, training gradually transitions to full generation optimization, refining responses and aligning them with domain-specific preferences. This staged adaptation balances domain knowledge acquisition with the preservation of general multimodal capabilities. Extensive experiments across specialized domains and general benchmarks validate the effectiveness of RCPA, establishing a practical pathway toward building high-performing and domain-adaptive VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。