分阶段优化让大模型更严格遵守复杂指令中的硬约束。
STAIF: A Stage-wise Optimization for Complex Instruction Following
- 分两阶段：先优化软约束，再用可验证奖励强化硬约束。
- 在3.1万条多约束指令上达到当前最优，且泛化能力强。
- 适合需要精准执行复杂规则的场景，如法律、医疗问答。
大语言模型在遵循包含多个显式约束的复杂指令时仍面临挑战。现有对齐方法（如DPO）通常优化整体奖励信号，往往忽视单个约束的严格满足，尤其在分布外或多重约束场景下。本文提出STAIF，一种分阶段优化框架，将主观（软）约束与客观可验证（硬）约束的对齐解耦。第一阶段采用带多个负样本的偏好优化，增强对软约束的敏感性；第二阶段使用可验证奖励强化学习（RLVR），强制满足硬约束。为支持该方法，构建了约3.1万条双语（英文/中文）高质量复杂多约束指令数据集STAINSTRUCT。大量分析验证了STAIF设计的有效性，在代表性基准上表现优于强基线，具备真实泛化能力。
原文摘要 · Abstract (English)
Following complex instructions with multiple explicit constraints remains a fundamental challenge for large language models (LLMs). Existing alignment methods, such as DPO, optimize holistic reward signals that often underemphasize strict satisfaction of individual constraints, particularly under out-of-distribution or multi-constraint settings. In this paper, we propose STAIF, a stage-wise optimization framework that decouples the alignment of subjective (soft) constraints from the optimization of objectively verifiable (hard) constraints. Stage 1 applies preference optimization with multiple negative samples to sharpen sensitivity to soft constraints, while Stage 2 applies Reinforcement Learning with Verifiable Rewards (RLVR) to enforce strict compliance with hard constraints. To support this method, we construct STAINSTRUCT, a high-quality bilingual (English, Chinese) dataset of approximately 31,000 complex multi-constraint instructions. Extensive analyses validate the design of STAIF and show state-of-the-art performance on representative benchmarks against strong baselines, as well as genuine generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。