测试大模型在细微语义变化下的指令遵循可靠性，发现性能可下降61.8%。
Revisiting the Reliability of Language Models in Instruction-Following
- 通过数据增强生成语义相近但表述不同的提示，评估模型一致性
- 20个私有与26个开源模型在细微修改下性能最高下降61.8%
- 提出新指标reliable@k和IFEval++基准，适合关注模型鲁棒性的研究者
先进大语言模型在IFEval等基准上已达到接近满分的指令遵循准确率。然而，这些高分未必代表真实场景下的可靠性，因用户常改变表达方式、上下文或任务表述。本文研究语义细微差异下的可靠性：模型在表达相似意图但细节不同的提示中是否保持一致表现。为此，我们引入新指标reliable@k，并构建自动化数据增强流水线生成高质量同类提示。基于此，我们构建了IFEval++用于系统评估。在20个私有和26个开源模型上，我们发现当前模型在语义细微变化下存在显著不可靠性——性能最高可下降61.8%。我们进一步分析该现象并探索三种改进方法。研究强调语义细微可靠性是实现更可信大模型行为的关键却未被充分重视的方向。代码与基准已公开：https://github.com/jianshuod/IFEval-pp。
原文摘要 · Abstract (English)
Advanced LLMs have achieved near-ceiling instruction-following accuracy on benchmarks such as IFEval. However, these impressive scores do not necessarily translate to reliable services in real-world use, where users often vary their phrasing, contextual framing, and task formulations. In this paper, we study nuance-oriented reliability: whether models exhibit consistent competence across cousin prompts that convey analogous user intents but with subtle nuances. To quantify this, we introduce a new metric, reliable@k, and develop an automated pipeline that generates high-quality cousin prompts via data augmentation. Building upon this, we construct IFEval++ for systematic evaluation. Across 20 proprietary and 26 open-source LLMs, we find that current models exhibit substantial insufficiency in nuance-oriented reliability -- their performance can drop by up to 61.8% with nuanced prompt modifications. What's more, we characterize it and explore three potential improvement recipes. Our findings highlight nuance-oriented reliability as a crucial yet underexplored next step toward more dependable and trustworthy LLM behavior. Our code and benchmark are accessible: https://github.com/jianshuod/IFEval-pp.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。