通过输入输出偏好优化，提升大模型复杂指令理解能力
IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization
- 提出IOPO方法，同时优化输入和输出偏好对
- 在复杂指令上比SFT、DPO分别提升8.15%和6.29%
- 适配高复杂度任务，适合智能体与自动化应用
大型语言模型（LLMs）在构建智能体与应用时，准确遵循指令的能力至关重要，而指令复杂性正快速上升。然而，复杂指令评估数据有限，且缺乏专门优化方法。为此，本文提出TRACE基准，包含12万条训练数据和1000条评估数据。同时提出IOPO（输入-输出偏好优化）对齐方法，兼顾输入与输出偏好对，使模型不仅能快速匹配响应偏好，还能精细捕捉指令偏好。在领域内与跨领域数据集上的大量实验表明，该方法相比SFT和DPO分别在领域内提升8.15%、2.18%，在跨领域提升6.29%、3.13%。
原文摘要 · Abstract (English)
In the realm of large language models (LLMs), the ability of models to accurately follow instructions is paramount as more agents and applications leverage LLMs for construction, where the complexity of instructions are rapidly increasing. However, on the one hand, there is only a certain amount of complex instruction evaluation data; on the other hand, there are no dedicated algorithms to improve the ability to follow complex instructions. To this end, this paper introduces TRACE, a benchmark for improving and evaluating the complex instructionfollowing ability, which consists of 120K training data and 1K evaluation data. Furthermore, we propose IOPO (Input-Output Preference Optimization) alignment method which takes both input and output preference pairs into consideration, where LLMs not only rapidly align with response preferences but also meticulously explore the instruction preferences. Extensive experiments on both in-domain and outof-domain datasets confirm the effectiveness of IOPO, showing 8.15%, 2.18% improvements on in-domain data and 6.29%, 3.13% on outof-domain data compared to SFT and DPO respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。