通过打包相似提示减少计算开销，显著提升大模型偏好训练效率
Preference Packing: Efficient Preference Optimization for Large Language Models
- 将相同输入的多个响应打包处理，减少重复计算
- 在文本和图文数据集上实现至少37%的训练时间降低
- 可与现有优化技术叠加使用，最高提速3.22倍
随着大语言模型规模持续扩大，资源高效训练优化技术愈发重要。目前预训练和监督微调中普遍采用批处理打包以实现资源高效训练。本文提出偏好打包（Preference Packing），用于提升使用同一输入不同响应数据的训练方法（如奖励模型或直接偏好优化DPO）的资源效率。该方法通过减少重复输入提示的注意力计算量并降低键值缓存内存占用，有效提升训练效率。我们在纯文本及包含图像的数据集上进行了实验，均实现了至少37%的训练时间缩减。值得注意的是，该方法可与批处理排序等现有优化技术结合使用，带来高达3.22倍的加速效果。
原文摘要 · Abstract (English)
Resource-efficient training optimization techniques are becoming increasingly important as the size of large language models (LLMs) continues to grow. In particular, batch packing is commonly used in pre-training and supervised fine-tuning to achieve resource-efficient training. We propose preference packing, a method to enhance resource efficiency in training techniques that use data with different responses for the same input prompt, such as reward models or Direct Preference Optimization (DPO). Preference packing improves resource efficiency by reducing the attention operations for duplicate input prompts and decreasing KV cache memory usage. We conducted experiments on text-only datasets and image-included datasets and achieved at least 37% reduction in training time. Notably, this method can be applied alongside existing optimization techniques such as batch sorting, resulting in a 3.22x speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。