用不确定性评估筛选高效数据，让大模型强化学习提速16倍
UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection
- 通过单次遍历估算不确定性，快速识别有价值数据
- 仅用10%数据训练，性能媲美全量数据，提速16倍
- 适合追求高效微调的大模型研究与工业落地
大语言模型的强化学习训练成本高昂，主要源于策略优化与评估所需的多次采样。受最近发展区（ZPD）理论启发，我们假设大模型在自身潜在理解范围内的数据上学习效果最佳。针对传统多采样方法计算开销大的问题，提出UFO-RL框架，采用计算高效的单次遍历不确定性估计，实现高达185倍的数据评估加速。该方法利用此指标筛选位于估计ZPD内的数据用于训练。实验表明，仅使用UFO-RL选出的10%数据进行训练，性能即可达到或超越全量数据训练，整体训练时间减少最多16倍，同时提升训练稳定性与泛化能力。UFO-RL为大模型强化学习微调提供了一种实用且高效的可扩展策略。
原文摘要 · Abstract (English)
Scaling RL for LLMs is computationally expensive, largely due to multi-sampling for policy optimization and evaluation, making efficient data selection crucial. Inspired by the Zone of Proximal Development (ZPD) theory, we hypothesize LLMs learn best from data within their potential comprehension zone. Addressing the limitation of conventional, computationally intensive multi-sampling methods for data assessment, we introduce UFO-RL. This novel framework uses a computationally efficient single-pass uncertainty estimation to identify informative data instances, achieving up to 185x faster data evaluation. UFO-RL leverages this metric to select data within the estimated ZPD for training. Experiments show that training with just 10% of data selected by UFO-RL yields performance comparable to or surpassing full-data training, reducing overall training time by up to 16x while enhancing stability and generalization. UFO-RL offers a practical and highly efficient strategy for scaling RL fine-tuning of LLMs by focusing learning on valuable data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。