解决大模型提示词模糊问题，提升指令执行稳定性
What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts
- 提出感知需求的提示优化方法，增强模型对缺失信息的理解
- 在模型或提示变化时，性能下降减少一半，平均提升4.8%
- 适合构建可靠LLM应用的研发团队使用
提示词不明确是与大模型交互中的常见挑战。本文深入分析该问题，发现尽管大模型默认可推断未指定要求（41.1%），但该行为脆弱：提示不明确时，模型或提示变更导致性能退化的可能性增加2倍，准确率下降有时超过20%。这种不稳定性使构建可靠大模型应用困难。此外，简单地完整指定所有要求也未必有效，因模型指令遵循能力有限且要求间可能冲突。标准提示优化器同样效果有限。为此，我们提出需求感知的提示优化机制，在基线基础上平均提升4.8%性能。同时倡导建立主动发现、评估与监控需求的系统化流程，以更好管理提示词不明确问题。
原文摘要 · Abstract (English)
Prompt underspecification is a common challenge when interacting with LLMs. In this paper, we present an in-depth analysis of this problem, showing that while LLMs can often infer unspecified requirements by default (41.1%), such behavior is fragile: Under-specified prompts are 2x as likely to regress across model or prompt changes, sometimes with accuracy drops exceeding 20%. This instability makes it difficult to reliably build LLM applications. Moreover, simply specifying all requirements does not consistently help, as models have limited instruction-following ability and requirements can conflict. Standard prompt optimizers likewise provide little benefit. To address these issues, we propose requirements-aware prompt optimization mechanisms that improve performance by 4.8% on average over baselines. We further advocate for a systematic process of proactive requirements discovery, evaluation, and monitoring to better manage prompt underspecification in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。