用词概率测量发现,提示语不同会让大模型输出差异显著。
Production and Perception in LLMs: A Token Probability Approach

- 通过直接测量词元概率,对比生成与感知提示下的模型输出
- 生成-感知距离是生成-生成距离的1.8倍,差异明显且可复现
- 提示语风格影响模型概率分布,尤其在序列开头最显著
语言生成与理解之间的不对称性在心理语言学中已有充分研究。大型语言模型(LLMs)是否具有功能上类似的区分仍属开放问题,尤其是因为它们在输入和输出处理中均依赖相同的机制(下一个词预测)。本探索性研究通过直接的词元概率测量而非元语言提示来操作生成-感知区别。使用基础版 Llama-3.1-8B 模型,在生成提示下生成诗歌,并对相同词元在重述生成提示和感知导向提示下重新评分。在包含四组生成提示与三组感知提示的扩展实验中,生成-感知距离始终显著高于生成-生成距离,各条件间无重叠,总体平均比值约为1.8。生成-生成对照中的近天花板相关性表明该效应特异于交际框架,而非提示表面变化。该现象在五种开源模型(Llama-3.1-8B、EuroLLM-9B、gemma-2-9b-it、Mistral-7B-Instruct-v0.3 和 Qwen2.5-7B-Instruct)中复现,涵盖基础模型与指令微调版本。时间分析显示,感知提示的影响在序列起始阶段最强,随着生成上下文积累而衰减,但衰减形态因提示对而异。这些发现表明,仅靠提示语框架即可在解码器仅架构中诱发生成-感知的概率分布差异。
原文摘要 · Abstract (English)
The asymmetry between language production and perception has been well-documented in psycholinguistics. Whether large language models (LLMs) exhibit a functionally analogous distinction remains an open question, particularly given that LLMs rely on the same underlying mechanism (next-token prediction) for both input and output processing. In this exploratory study, we operationalize the production-perception distinction through direct token probability measurements rather than metalinguistic prompting. Using the base Llama-3.1-8B model, we generated poems under a production prompt and re-scored the same tokens under both rephrased production prompts and perception-oriented prompts. Across an extended experiment with four production and three perception prompts, production-perception distances consistently and substantially exceeded production-production distances, with non-overlapping ranges across conditions and an overall average ratio of approximately 1.8. Near-ceiling correlations in the production-production control confirm that the effect is specific to communicative framing rather than prompt surface variation, and we show the effect replicates across five open-weight models (Llama-3.1-8B, EuroLLM-9B, gemma-2-9b-it, Mistral-7B-Instruct-v0.3, and Qwen2.5-7B-Instruct), spanning both base and instruction-tuned variants. Temporal analysis revealed that the perception prompt exerts its strongest influence at the beginning of the sequence, with divergence decaying as generated context accumulates, though the specific shape of this decay varies across prompt pairs. These findings suggest that prompt framing alone induces a production-perception distinction in LLM probability distributions, even within a decoder-only architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。