提升大模型对柔性约束的遵循能力，方法可复现。
Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models
- 自动构建高质量数据集，利用正负样本优化模型
- 采用课程学习策略，按约束数量分阶段训练
- 在多个任务上验证有效性，适合需要精准指令执行的场景
大语言模型遵循多约束指令至关重要，但对柔性约束的处理仍属空白。为此，我们设计了一套自动化流程，生成高质量输出数据集。利用数据构建过程中产生的正负样本，采用直接偏好优化（DPO）进行训练。同时，根据约束数量设置课程学习策略，分阶段提升模型能力。实验评估表明该方法显著增强模型对柔性约束的遵循能力，并分析了关键影响因素。代码与数据集已开源：https://github.com/Rainier-rq/FollowSoftConstraint。
原文摘要 · Abstract (English)
It is crucial for large language models (LLMs) to follow instructions that involve multiple constraints. However, it is an unexplored area to enhance LLMs' ability to follow soft constraints. To bridge the gap, we initially design a pipeline to construct datasets with high-quality outputs automatically. Additionally, to fully utilize the positive and negative samples generated during the data construction process, we choose Direct Preference Optimization (DPO) as the training method. Furthermore, taking into account the difficulty of soft constraints indicated by the number of constraints, we design a curriculum learning training paradigm based on the constraint quantity. We experimentally evaluate the effectiveness of our methods in improving LLMs' soft constraint following ability and analyze the factors driving the improvements.The datasets and code are publicly available at https://github.com/Rainier-rq/FollowSoftConstraint.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。