不同提示格式可让模型性能波动40%，影响代码翻译效果
Does Prompt Formatting Have Any Impact on LLM Performance?
- 用纯文本、Markdown、JSON等格式重写相同提示内容
- GPT-3.5-turbo在代码翻译任务中性能差高达40%
- 大模型如GPT-4对格式变化更鲁棒，适合作业级应用
在大型语言模型（LLMs）领域，提示优化对模型表现至关重要。尽管已有研究关注提示重述、上下文学习、思维链等技术，但对提示模板敏感性的理解仍有限。本文系统考察了不同提示模板对模型性能的影响。将相同内容以纯文本、Markdown、JSON和YAML等人类可读格式重写，并在自然语言推理、代码生成和翻译任务中评估OpenAI的GPT系列模型表现。实验显示，在代码翻译任务中，GPT-3.5-turbo的表现随模板格式变化最大可达40%；而更大模型如GPT-4则表现出更强的鲁棒性。分析表明，应重新审视固定提示模板的使用，因为格式差异可能显著影响模型输出。
原文摘要 · Abstract (English)
In the realm of Large Language Models (LLMs), prompt optimization is crucial for model performance. Although previous research has explored aspects like rephrasing prompt contexts, using various prompting techniques (like in-context learning and chain-of-thought), and ordering few-shot examples, our understanding of LLM sensitivity to prompt templates remains limited. Therefore, this paper examines the impact of different prompt templates on LLM performance. We formatted the same contexts into various human-readable templates, including plain text, Markdown, JSON, and YAML, and evaluated their impact across tasks like natural language reasoning, code generation, and translation using OpenAI's GPT models. Experiments show that GPT-3.5-turbo's performance varies by up to 40\% in a code translation task depending on the prompt template, while larger models like GPT-4 are more robust to these variations. Our analysis highlights the need to reconsider the use of fixed prompt templates, as different formats can significantly affect model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。