小模型中，监督微调仍是主引擎，强化学习优化收益有限。
An Empirical Study of SFT-DPO Interaction and Parameterization in Small Language Models
- 对比SFT、DPO及混合训练，评估小模型表现
- 全参数微调优于低秩适配，且无加速效果
- 偏好优化仅在目标匹配时有小幅提升
直接偏好优化（DPO）常在监督微调（SFT）后用于对齐语言模型，但在小型骨干网络和有限数据下的实证行为尚不明确。本研究系统比较了仅SFT、仅DPO、SFT→DPO分阶段训练，以及全参数微调（FFT）与LoRA在GPT-2规模解码器上的表现，评估任务包括改写检测和莎士比亚十四行诗续写。结果显示，DPO在强SFT基础上带来微小且任务相关的增益；当偏好构建与监督目标高度一致时，DPO可在无预热的情况下达到竞争性SFT精度。相反，参数化方式起主导作用：在相同训练深度下，FFT始终优于LoRA，且在本实验硬件上LoRA未减少实际耗时。这些发现表明，在此小规模场景下，监督式全参数微调仍是主要性能提升手段，而偏好优化与低秩适配带来的边际收益有限。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) is widely used after supervised fine-tuning (SFT) to align language models, yet empirical behavior under small backbones and modest data is under-specified. We systematically compare SFT-only, DPO-only, and staged SFT-to-DPO training alongside full fine-tuning (FFT) versus LoRA on a GPT-2-scale decoder, evaluating paraphrase detection and Shakespearean sonnet continuation. DPO yields small, task-dependent gains over strong SFT and can match competitive SFT accuracy without a warm start when the preference construction closely parallels the supervised objective. In contrast, parameterization dominates: FFT consistently outperforms LoRA at matched training depth, and LoRA does not reduce wall-clock time on our hardware. These findings indicate that, in this small-scale regime, supervised full-parameter adaptation remains the primary performance lever, while preference optimization and low-rank adaptation provide limited marginal returns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。