在计算资源有限时，如何高效选择训练数据以提升大模型微调效果。
Compute-Constrained Data Selection
- 设计成本感知的选数据策略，平衡选数据与训练的开销
- 发现廉价选数据方法在多数情况下更优，甚至优于强力方法
- 明确不同方法所需的模型尺寸比例，指导实际应用
数据选择可减少微调大模型所需的数据量，但其效果与计算开销正相关。针对计算受限的微调场景，本文考虑同时预算数据选择和训练的开销。我们形式化了带成本意识的效用函数，将数据选择建模为初始选择成本与训练收益之间的权衡。通过在多个任务上系统性实验，调整微调词数、模型规模和数据选择计算资源，发现许多强大的数据选择方法几乎从不具有计算最优性，而更低成本的方法在理论和实证层面均占优。对于计算最优的训练，困惑度和梯度数据选择分别要求训练模型与选数据模型大小比达到5倍和10倍。
原文摘要 · Abstract (English)
Data selection can reduce the amount of training data needed to finetune LLMs; however, the efficacy of data selection scales directly with its compute. Motivated by the practical challenge of compute-constrained finetuning, we consider the setting in which both the cost of selecting data and training are budgeted for. We first formalize the problem of data selection with a cost-aware utility function, and model the data selection problem as trading off initial-selection cost for training gain. We run a comprehensive sweep of experiments across multiple tasks, varying compute budget by scaling finetuning tokens, model sizes, and data selection compute. Interestingly we find that many powerful data selection methods are almost never compute-optimal, and that cheaper data selection alternatives dominate both from a theoretical and empirical perspective. For compute-optimal training, we find that perplexity and gradient data selection require training-to-selection model size ratios of 5x and 10x, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。