用重要性加权改进数据筛选的微调，让监督学习更像强化学习。
Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)
- 在筛选数据上做监督微调，本质是强化学习的下界优化。
- 改进方法在AIME 2024上达到66.7%准确率，优于传统SFT。
- 实现简单，适合想提升微调效果的研究者和工程师。
在经过筛选的数据上进行行为克隆（BC）是大语言模型监督微调（SFT）及控制策略模仿学习的主流方法。本文揭示该策略与强化学习（RL）理论之间的联系，指出SFT可被理解为在稀疏奖励设置下最大化RL目标的下界，解释了其常表现出色的原因。基于此视角，我们提出一种小修改：重要性加权监督微调（iw-SFT），它优化了更紧的下界，并在筛选数据上表现优于传统SFT。该方法易于实现，还可推广至质量评分数据训练。在大语言模型和连续控制任务中，其性能媲美先进强化学习算法，例如在AIME 2024数据集上达到66.7%准确率。
原文摘要 · Abstract (English)
Behavior Cloning (BC) on curated (or filtered) data is the predominant paradigm for supervised fine-tuning (SFT) of large language models; as well as for imitation learning of control policies. Here, we draw on a connection between this successful strategy and the theory and practice of finding optimal policies via Reinforcement Learning (RL). Building on existing literature, we clarify that SFT can be understood as maximizing a lower bound on the RL objective in a sparse reward setting. Giving support to its often observed good performance. From this viewpoint, we realize that a small modification to SFT leads to an importance weighted variant that behaves closer to training with RL as it: i) optimizes a tighter bound to the RL objective and, ii) can improve performance compared to SFT on curated data. We refer to this variant as importance weighted supervised fine-tuning (iw-SFT). We show that it is easy to implement and can be further generalized to training with quality scored data. The resulting SFT variants are competitive with more advanced RL algorithms for large language models and for training policies in continuous control tasks. For example achieving 66.7% on the AIME 2024 dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。