将大模型分成快思考与慢思考两系统,高效适配不同推理需求。
LoRA-PAR: A Flexible Dual-System LoRA Partitioning Approach to Efficient LLM Fine-Tuning
- 按任务类型划分数据与参数,分属快思与慢思系统
- 双阶段训练:监督微调提升直觉,强化学习增强逻辑推理
- 参数量更少却性能不输顶尖方法,适合资源受限场景
如 DeepSeek-R1 和 OpenAI-O1 等大规模生成模型在链式思维(CoT)推理中表现优异,但提升性能通常需海量数据、巨大模型规模及全参数微调。尽管参数高效微调(PEFT)可降低成本,多数现有方法仅聚焦领域迁移或层间分配,未针对不同响应需求明确匹配数据与参数。受《思考,快与慢》启发,我们类比系统1(快速、直觉、自动)与系统2(缓慢、分析、深思)的思维模式,提出 LoRA-PAR:一种双系统 LoRA 框架,根据系统1或系统2的需求对数据与参数进行分区。通过多模型角色扮演与投票分类任务数据,并基于重要性评分划分参数。采用两阶段微调策略:先用监督微调(SFT)训练系统1任务以增强知识与直觉,再用强化学习(RL)优化系统2任务以强化深度逻辑推理。大量实验表明,该策略在降低活跃参数使用的同时,性能达到或超越当前最先进 PEFT 基线。
原文摘要 · Abstract (English)
Large-scale generative models like DeepSeek-R1 and OpenAI-O1 benefit substantially from chain-of-thought (CoT) reasoning, yet pushing their performance typically requires vast data, large model sizes, and full-parameter fine-tuning. While parameter-efficient fine-tuning (PEFT) helps reduce cost, most existing approaches primarily address domain adaptation or layer-wise allocation rather than explicitly tailoring data and parameters to different response demands. Inspired by "Thinking, Fast and Slow," which characterizes two distinct modes of thought-System 1 (fast, intuitive, often automatic) and System 2 (slower, more deliberative and analytic)-we draw an analogy that different "subregions" of an LLM's parameters might similarly specialize for tasks that demand quick, intuitive responses versus those requiring multi-step logical reasoning. Therefore, we propose LoRA-PAR, a dual-system LoRA framework that partitions both data and parameters by System 1 or System 2 demands, using fewer yet more focused parameters for each task. Specifically, we classify task data via multi-model role-playing and voting, and partition parameters based on importance scoring, then adopt a two-stage fine-tuning strategy of training System 1 tasks with supervised fine-tuning (SFT) to enhance knowledge and intuition and refine System 2 tasks with reinforcement learning (RL) to reinforce deeper logical deliberation next. Extensive experiments show that the two-stage fine-tuning strategy, SFT and RL, lowers active parameter usage while matching or surpassing SOTA PEFT baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。