提示词和查询方式显著影响大模型在低数据下的预测表现。
Elicitation Matters: How Prompts and Query Protocols Shape LLM Surrogates under Sparse Observations
- 用结构化提示词作为先验，提升模型可信度
- 逐点与联合查询产生不同信念，顺序影响置信度变化
- 适合优化算法设计者和大模型应用研究者
大语言模型越来越多地被用作低数据优化的代理模型，但其面向优化器的预测结果及其不确定性仍不清晰。本文研究了在稀疏观测下从大模型中获取的代理信念，发现其强烈依赖于提示文本和查询协议。提出一种不确定性对齐准则，用于衡量模型不确定性是否反映样本一致函数间的残余模糊性。在控制推理任务与贝叶斯优化实验中，发现结构化提示词可作为有效先验，点式与联合查询导致不同信念，且序列证据引发非单调、顺序敏感的置信度更新。这些效应显著影响下游的采样决策与累计损失，表明提示获取协议是大模型代理模型的重要组成部分，而非简单的格式细节。
原文摘要 · Abstract (English)
Large language models are increasingly used as surrogate models for low-data optimization, but their optimizer-facing prediction and its uncertainty remain poorly understood. We study the surrogate belief elicited from an LLM under sparse observations, showing that it depends strongly on prompt text and query protocol. We introduce an uncertainty-alignment criterion that measures whether model uncertainty tracks residual ambiguity among sample-consistent functions. Across controlled inference tasks and Bayesian optimization studies, we find that structural prompts act as effective priors, POINTWISE and JOINT querying induce different beliefs, and sequential evidence leads to non-monotonic, order-sensitive confidence updates. These effects change downstream acquisition decisions and regret, showing that elicitation protocol is part of the LLM surrogate specification, not a formatting detail.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。