通过共享前缀加速偏好优化训练，提升效率不降性能。
Accelerating Direct Preference Optimization with Prefix Sharing
- 将选择与拒绝响应合并处理，共享相同前缀以减少重复计算。
- 在主流DPO数据集上训练吞吐量提升1.1到1.5倍，收敛不受影响。
- 适用于各类模型规模，尤其适合长提示任务，开源代码可复现。
离线成对偏好优化算法已成为在偏好数据上微调的热门方法，在多种任务中表现优于传统监督微调。然而，传统实现常因冗余计算导致效率低下，尤其在长共享提示任务中更为明显。本文提出偏好微调中的前缀共享技术，将选择与拒绝响应作为单一序列处理，共享前缀部分。为防止跨响应信息泄露，采用自定义块稀疏注意力掩码。该方法在主流DPO数据集上实现1.1至1.5倍的训练吞吐量提升，且不影响收敛性。结合序列打包后，仍保持1.3至1.6倍稳定加速，即使在较短序列数据集上也有效。尽管聚焦于直接偏好优化（DPO），本方法可推广至其他成对偏好微调方法。通过提升计算效率，使基于偏好的微调更适用于广泛的应用场景与模型规模。代码已开源：https://github.com/frankxwang/dpo-prefix-sharing。
原文摘要 · Abstract (English)
Offline paired preference optimization algorithms have become a popular approach for fine-tuning on preference data, outperforming traditional supervised fine-tuning in various tasks. However, traditional implementations often involve redundant computations, especially for tasks with long shared prompts. We introduce prefix sharing for preference tuning, a novel technique that processes chosen and rejected responses as one sequence with a shared prefix. To prevent cross-response contamination, we use a custom block-sparse attention mask. Our method achieves $1.1$-$1.5\times$ improvement in training throughput on popular DPO datasets, without any effect on convergence. When combined with sequence packing, we observe consistent $1.3$-$1.6\times$ speedups, benefiting even datasets with smaller sequence lengths. While we focus on Direct Preference Optimization (DPO), our approach is applicable to other paired preference tuning methods. By enhancing computational efficiency, our work contributes to making preference-based fine-tuning more accessible for a wider range of applications and model sizes. We open-source our code at https://github.com/frankxwang/dpo-prefix-sharing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。