对比两种模型微调方法，发现不同条件下各有优势。
Learning to Choose or Choosing to Learn: Best-of-N vs. Supervised Fine-Tuning for Bit String Generation
- 用监督微调训练新预测器，或用奖励模型从原始模型生成中选优。
- 可实现时，监督微调收敛更快；不可实现时，Best-of-N表现更优。
- 适合研究模型适配策略，尤其关注生成质量与效率的场景。
以位串生成问题为案例，理论比较两种标准的大语言模型任务适配方法。第一种是监督微调，即在优质生成结果上训练新的下一个词预测器；第二种是Best-of-N（BoN），通过训练奖励模型从原始基础模型生成的结果中筛选优质响应。若学习设定可实现，监督微调在收敛速度上对响应长度的依赖更优，表现更佳；若可实现性不成立，则根据失败模式，BoN可在n或对响应长度的依赖上获得更好的收敛速率。
原文摘要 · Abstract (English)
Using the bit string generation problem as a case study, we theoretically compare two standard methods for adapting large language models to new tasks. The first, referred to as supervised fine-tuning, involves training a new next token predictor on good generations. The second method, Best-of-N, trains a reward model to select good responses from a collection generated by an unaltered base model. If the learning setting is realizable, we find that supervised fine-tuning outperforms BoN through a better dependence on the response length in its rate of convergence. If realizability fails, then depending on the failure mode, BoN can enjoy a better rate of convergence in either n or a rate of convergence with better dependence on the response length.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。