在强预训练模型中，随机采样比主动选择更有效。
Random Is Hard to Beat: Active Selection in online DPO with Modern LLMs
- 用不确定性驱动的主动学习对比随机采样
- 主动学习在代理胜率上提升微弱，但能力下降
- 适合关注效率与简单策略的实践者
现代大语言模型继承了网络规模预训练带来的强大先验，限制了后训练数据选择策略的提升空间。尽管主动偏好学习（APL）旨在优化在线直接偏好优化（DPO）中的查询效率，但策略池中固有的丰富性使得简单随机采样成为出人意料的强大基线。我们在无害性、帮助性和指令遵循设置下，使用奖励模型和大模型作为裁判代理评估不确定性驱动的APL与随机采样的表现。结果发现，APL在代理胜率上的提升微乎其微。关键的是，我们观察到胜率提升的同时标准基准测试能力反而下降，表明存在解耦现象。APL未能缓解这种能力坍塌，也未显著优于随机采样减少方差。我们的研究暗示，在强预训练先验的场景下，主动选择带来的计算开销难以抵消随机采样提供的‘低成本多样性’。代码已公开于https://github.com/BootsofLagrangian/random-vs-apl。
原文摘要 · Abstract (English)
Modern LLMs inherit strong priors from web-scale pretraining, which can limit the headroom of post-training data-selection strategies. While Active Preference Learning (APL) seeks to optimize query efficiency in online Direct Preference Optimization (DPO), the inherent richness of on-policy candidate pools often renders simple Random sampling a surprisingly formidable baseline. We evaluate uncertainty-based APL against Random across harmlessness, helpfulness, and instruction-following settings, utilizing both reward models and LLM-as-a-judge proxies. We find that APL yields negligible improvements in proxy win-rates compared to Random. Crucially, we observe a dissociation where win-rate improves even as general capability -- measured by standard benchmarks -- degrades. APL fails to mitigate this capability collapse or reduce variance significantly better than random sampling. Our findings suggest that in the regime of strong pre-trained priors, the computational overhead of active selection is difficult to justify against the ``cheap diversity'' provided by simple random samples. Our code is available at https://github.com/BootsofLagrangian/random-vs-apl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。