大规模实验揭示提示格式、指令数和上下文长度对大模型遵循指令与幻觉的影响
Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models
- 通过控制变量实验，测试四种格式下指令数量与上下文长度对模型表现的影响
- 指令超过80条时，所有模型的正确响应率降至零；上下文超64k tokens后召回率急剧下降
- 模型拒绝回答的比例在接近上下文极限时飙升至79%-90%，远超幻觉或迎合现象
实践者在提示设计中面临三大无实证支持的决策：指令与上下文的格式（如Markdown、纯文本、散文或表格）、系统提示中可容纳的并行指令数量上限，以及模型能有效处理的上下文长度。我们基于一个固定种子生成、不含污染的合成语料库（Book of Veyra，包含8,780个唯一命名实体）进行了两项受控实验，覆盖五个模型。实验1（每模型960次调用）考察指令数从10增至160时，不同格式及系统提示与用户轮次位置对指令遵循的影响。结果表明，所有模型在指令数达80时正确响应率归零，位置影响至少与格式相当，且方向因模型而异；无模型显示Markdown有优势，其中一个350亿参数模型反而偏好纯文本。实验2（每模型5,520次调用）评估了2k至512k token上下文跨度下的召回准确率、虚假前提迎合及缺失事实编造情况。召回率在64-128k token内保持高位，之后急剧下降且依赖格式：某模型在128k token时准确率差异达48分。编造行为未出现（0/5,760次探测），迎合现象也极低（≤8.3%）。但接近各模型上下文上限时，拒绝回答比例从0%升至79%-90%，区别于迎合或编造。预设格式顺序无效，且格式带来22%-37%的令牌开销，进一步改变最优选择。数据集与工具已开源（VeyraBench）：https://github.com/iNetanel/veyrabench
原文摘要 · Abstract (English)
Practitioners make three prompt-design decisions with almost no controlled evidence behind them: how to format instructions and context (markdown, plain text, prose, or tabular), how many simultaneous instructions a system prompt can carry before compliance degrades, and how much context a model can hold before recall and honesty degrade. We report two controlled experiments crossing all three factors on one held, contamination-free synthetic corpus (the "Book of Veyra," 8,780 uniquely-named entities, deterministically regenerable from a fixed seed), evaluated across five models. Experiment 1 (960 calls/model) measures instruction-following decay as rule count N grows from 10 to 160, crossed with four formats and system-prompt vs. user-turn placement. Perfect-response rate collapses to zero by N=80 for every model, format, and placement. Placement produces effects at least as large as format at N=160 in most models, but the direction is model-specific. No model shows a reliable markdown advantage; one 35B model favors plain text instead. Experiment 2 (5,520 calls/model) measures recall accuracy, false-premise sycophancy, and absent-fact fabrication across a 2k-to-512k-token context ladder in the same four formats. Recall stays near ceiling through 64-128k tokens, then degrades sharply and format-dependently: one model's accuracy spread reaches 48 points at 128k tokens. Fabrication never occurs (0/5,760 probes), and sycophancy stays negligible (<=8.3%). What rises sharply near each model's context ceiling is outright refusal to answer (0% to 79-90%), distinct from sycophancy or fabrication. Neither pre-registered format ordering holds, and token overhead (+22% to +37% over plain text) further changes which format is preferable where accuracy spread is genuine. We release the full harness, corpus generator, and raw results (VeyraBench): https://github.com/iNetanel/veyrabench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。