大模型能单条听话，但多条一起听就崩，6条以上成功率骤降。
Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction
- 用规则验证器系统测试多个约束同时满足的能力，避免人工评分偏差。
- 15个模型在8个约束下整体通过率仅5.7%，远低于单约束41%的通过率。
- 结构类约束比词汇类更难维持，失败多因输出特征共享而非相互干扰。
大语言模型在遵循单一约束时表现良好，但在需同时满足多个明确约束（如推理结构、安全边界、输出格式）的组合场景中表现仍不清晰：性能下降速度如何、受何因素影响、能否缓解？我们提出约束饱和评估（CSE），一个可程序生成的基准，系统性地改变同时约束数量（k），每个约束由确定性规则验证器评分，不依赖任何LLM判断：涵盖15个模型、36种约束类型，共369,753次验证，k值从1到12。研究发现：第一，单约束通过率渐进下降，但所有约束同时满足的概率急剧崩溃——某模型在k=8时单约束通过率为41%，但全部通过率仅为5.7%；第二，不同约束退化程度不同：结构性约束每增加一条损失的能力是词汇类的2倍，其衰减由理解-维持差距决定，即需要持续追踪的约束易失效，而二元决策类则较稳定；第三，失败几乎独立，导致累积效应呈乘法式恶化；残余关联主要来自共享输出特征，例如错误的句子数会导致所有读取该数值的约束都失败。可靠指令遵循能力在5-6个以上约束时失效：最强模型在7个约束时通过率低于50%，其余12个模型在3个或更少约束时即低于50%。
原文摘要 · Abstract (English)
Large language models are increasingly deployed in settings that require simultaneous adherence to multiple explicit constraints - reasoning structure, safety boundaries, output schemas. Individual constraints are handled proficiently, but the compositional regime, where many must hold jointly, remains poorly characterized: how rapidly does performance degrade, what governs the degradation, and can the collapse be mitigated? We introduce Constraint Saturation Evaluation (CSE), a procedurally generated benchmark that systematically varies the number of simultaneous constraints (k), with every constraint scored by a deterministic, rule-based verifier and zero LLM-judge involvement: 15 models, 36 constraint types, 369,753 checks at k=1-12. Three findings emerge. First, per-constraint pass rate decays gradually and predictably, while the chance of satisfying all k constraints collapses - a model passing individual constraints at ~41% at k=8 succeeds on all eight just 5.7% of the time. Second, constraints do not degrade equally: structural constraints lose 2x more baseline capability per added constraint than lexical ones, ordered by a comprehension-maintenance gap that separates constraints requiring sustained tracking from binary decisions immune to composition. Third, failures are nearly independent, which is what makes the accumulation multiplicative; the residual coupling that does exist tracks shared output features rather than pairwise interference - a wrong sentence count fails every constraint that reads it. Reliable instruction following breaks down beyond 5-6 simultaneous constraints: probe-level success falls below 50% at 7 constraints for the strongest model, and at 3 or fewer for 12 of 15.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。