提升AI模型对新指令约束的泛化能力，解决其盲目遵循旧规则的问题。
Generalizing Verifiable Instruction Following
- 设计可验证的约束模块，通过强化学习优化模型对指令的准确执行。
- 在58个新约束上测试，现有模型泛化能力差，仅能正确响应约30%的未见约束。
- 提供新数据集、验证函数和训练模板，助力研究者提升模型指令遵循能力。
人类与人工智能交互的成功关键在于语言模型能否精确遵循指令。许多指令包含可验证的输出约束，如“仅回答是或否”或“至少三次提及‘abrakadabra’”。尽管当前最强模型仍难以满足这些要求，我们发现它们在基准测试中使用的少量可验证约束上严重过拟合，难以泛化到未见的约束。为此，我们提出新基准IFBench，用于评估模型在58个全新、多样且具有挑战性的跨领域可验证约束上的泛化表现。我们还系统分析了模型训练方法，设计了精准的约束验证模块，并证明基于可验证奖励的强化学习（RLVR）显著提升指令遵循能力。除IFBench外，我们还公开29个手工标注的训练约束、验证函数、训练提示和代码。
原文摘要 · Abstract (English)
A crucial factor for successful human and AI interaction is the ability of language models or chatbots to follow human instructions precisely. A common feature of instructions are output constraints like ``only answer with yes or no" or ``mention the word `abrakadabra' at least 3 times" that the user adds to craft a more useful answer. Even today's strongest models struggle with fulfilling such constraints. We find that most models strongly overfit on a small set of verifiable constraints from the benchmarks that test these abilities, a skill called precise instruction following, and are not able to generalize well to unseen output constraints. We introduce a new benchmark, IFBench, to evaluate precise instruction following generalization on 58 new, diverse, and challenging verifiable out-of-domain constraints. In addition, we perform an extensive analysis of how and on what data models can be trained to improve precise instruction following generalization. Specifically, we carefully design constraint verification modules and show that reinforcement learning with verifiable rewards (RLVR) significantly improves instruction following. In addition to IFBench, we release 29 additional new hand-annotated training constraints and verification functions, RLVR training prompts, and code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。