arXiv:2506.18048cs.CV2025-06

用分阶段训练提升小模型推理能力,效果接近大模型。

CLGRPO: Reasoning Ability Enhancement for Small VLMs

  • 分四阶段训练:注入知识、对齐格式、强化推理、压缩训练空间。
  • 1B小模型在数据集上准确率提升2.77点,召回率升0.69点。
  • 适合资源受限场景下追求高性能小模型的开发者使用。

小型视觉语言模型(SVLMs)通常指参数量不超过20亿的模型,因其低功耗和低成本具备高商业价值。然而其推理能力受限于参数规模。本文提出一种后训练优化范式——增量训练策略,以增强小模型推理能力。首先构建自监督思维链(COT)数据生成系统,利用70亿及以上参数的LVLM将原始数据转化为COT数据。该策略包含四个阶段:第一阶段通过监督微调(SFT)在COT数据上注入领域知识;第二阶段以格式奖励为约束进行少量组相对策略优化(GRPO),对齐数据格式;第三阶段在格式与准确率双重约束下应用GRPO,显著提升推理能力;第四阶段提出ClipLow GRPO(CLGRPO),限制训练过程中的捕获空间,缓解小模型容量不足问题。在抽象语义识别数据集EMOSet-118K上进行大量对比与消融实验。结果表明,该方法使1B SVLM的推理能力显著提升,相比原始数据微调基线,准确率提高2.77,召回率提高0.69,性能接近8B模型。

原文摘要 · Abstract (English)

Small Vision Language Models (SVLMs) generally refer to models with parameter sizes less than or equal to 2B. Their low cost and power consumption characteristics confer high commercial value. However, their reasoning abilities are limited by the number of parameters. To address this issue, this paper proposes a post-training optimization paradigm called the Incremental Training Strategy to enhance the reasoning ability of SVLMs. Firstly, we constructed a Self-Supervised Chain-of-Thought (COT) Data Construction System, which leverages multiple LVLMs with 7B parameters or more to transform original data into COT data in a self-supervised manner. Our proposed Incremental Training Strategy consists of four stages. Stage 1 injects domain knowledge by performing Supervised Fine-Tuning (SFT) to the pretrained model on the COT data. Stage 2 aligns the COT data format by conducting a small amount of Group Relative Policy Optimization (GRPO) training constrained only by format rewards on the COT data. Stage 3 enhances reasoning ability by applying GRPO training on the COT data with constraints on both format and accuracy rewards. The resulting model shows significant improvement compared to the baseline. Stage 4 addresses the limited capacity of the SVLMs and the weak ability to capture complex patterns by proposing ClipLow GRPO (CLGRPO) to constrain the capture space of the training process. We conducted extensive comparative and ablation experiments on the abstract semantic recognition dataset EMOSet-118K. Experimental results demonstrate that our method significantly improves the reasoning ability of 1B SVLM. Compared to the baseline model fine-tuned on the original data, accuracy increased by 2.77 and recall by 0.69, achieving performance comparable to that of 8B models.

小模型推理增强视觉语言模型训练策略

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。