对比5种提示鲁棒性方法，发现不同模型对格式变化的敏感度差异。
When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs
- 统一框架评估5种提升提示鲁棒性的方法
- 在52个任务上测试8个模型，涵盖多种分布偏移
- 适用于希望提升大模型实际应用稳定性的开发者
大型语言模型对提示词的细微非语义变化极为敏感。本文首次在统一实验框架下系统评估了5种提升提示鲁棒性的方法。我们在来自Llama、Qwen和Gemma系列的8个模型上，基于Natural Instructions数据集中的52项任务进行基准测试。评估覆盖了微调与上下文学习两类鲁棒性方法，并检验其在多种分布偏移下的泛化能力。最后，我们还将分析扩展至GPT-4.1和DeepSeek V3，评估前沿模型对格式扰动的鲁棒性。研究结果为不同方法的有效性提供了可操作的见解，助力从业者在真实场景中实现稳定可靠的LLM表现。代码已开源：https://github.com/AIRI-Institute/when-punctuation-matters。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are highly sensitive to subtle, non-semantic variations in prompt phrasing and formatting. In this work, we present the first systematic evaluation of 5 methods for improving prompt robustness within a unified experimental framework. We benchmark these techniques on 8 models from Llama, Qwen and Gemma families across 52 tasks from Natural Instructions dataset. Our evaluation covers robustness methods from both fine-tuned and in-context learning paradigms, and tests their generalization against multiple types of distribution shifts. Finally, we extend our analysis to GPT-4.1 and DeepSeek V3 to assess frontier models' current robustness to format perturbations. Our findings offer actionable insights into the relative effectiveness of these robustness methods, enabling practitioners to make informed decisions when aiming for stable and reliable LLM performance in real-world applications. Code: https://github.com/AIRI-Institute/when-punctuation-matters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。