首次量化提示词中每个词的重要性,揭示有效提示的底层机制。
Which Words Matter Most in Zero-Shot Prompts?
- 通过替换、删除等扰动实验,系统评估提示词中每个词的作用。
- 数学任务重'逐步思考',推理任务重'思考',名词最关键。
- 发现提示词效果与模型能力反相关,适合提升弱模型表现。
尽管零样本指令提示如'让我们逐步思考'显著提升了大语言模型性能,但一个根本问题仍未解决:哪些具体词语驱动了其显著效果?我们提出ZIP评分(零样本扰动重要性),首个通过受控扰动(同义词替换、共下位词替换、策略性删除)量化指令提示中单个词语重要性的方法。在四个主流模型、七种广泛使用提示和多个任务领域上的分析揭示四大发现:(1) 任务特异性词序存在,数学问题优先'逐步思考',推理任务偏好'思考';(2) 专有模型比开源模型更符合人类直觉;(3) 名词占据重要性排名主导地位,始终是多数显著词语;(4) 词语重要性与模型性能呈负相关,提示影响最大时正是模型最薄弱处。除揭示这些模式外,我们还建立首个提示可解释性基准,包含20个预设关键词的验证提示,其中ZIP准确率达90%,远超LIME的60%。研究推动提示科学进步,为提示工程提供实用指导,深化对大模型中词汇层级效应的理论理解。
原文摘要 · Abstract (English)
While zero-shot instructional prompts like "Let's think step-by-step" have revolutionized Large Language Model performance, a fundamental question remains unanswered: which specific words drive their remarkable effectiveness? We introduce the ZIP score (Zero-shot Importance of Perturbation), the first systematic method to quantify individual word importance in instructional prompts through controlled perturbations including synonym replacement, co-hyponym substitution, and strategic removal. Our analysis across four flagship models, seven widely-adopted prompts, and multiple task domains reveals four key findings: (1) Task-specific word hierarchies exist where mathematical problems prioritize "step-by-step" while reasoning tasks favor "think"; (2) Proprietary models show superior alignment with human intuitions compared to open-source alternatives; (3) Nouns dominate importance rankings, consistently representing the majority of significant words; and (4) Word importance inversely correlates with model performance, indicating prompts have greatest impact where models struggle most. Beyond revealing these patterns, we establish the first ground-truth benchmark for prompt interpretability through 20 validation prompts with predetermined key words, where ZIP achieves 90% accuracy versus LIME's 60%. Our findings advance prompt science, the study of how language shapes model behavior, providing both practical insights for prompt engineering and theoretical understanding of word-level effects in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。