大模型难以按指定概率分布生成答案，连简单二选一都做不好。
Failure to Mix: Large language models struggle to answer according to desired probability distributions
- 要求模型按指定概率输出，但实际几乎只选最高概率项
- 要输出'1'49%时间，结果'0'出现近100%
- 连模型原有偏好都被无视，暴露生成机制缺陷
科学创意生成与选择需要按照目标概率分布进行探索。然而，当前AI评估基准都有确定的正确答案，通过这些基准对大语言模型（LLMs）进行强化学习训练，反而抑制了概率性探索。我们系统性地测试了现代大模型在遵循简单概率分布方面的表现，发现所有被测模型均严重偏离目标分布。例如，要求模型以49%的概率输出'1'，实际'0'的输出频率接近100%。这种近乎阶跃函数的行为表明，即使存在显著的内在模型偏见，模型仍会几乎唯一地生成概率略高的输出。
原文摘要 · Abstract (English)
Scientific idea generation and selection requires exploration following a target probability distribution. In contrast, current AI benchmarks have objectively correct answers, and training large language models (LLMs) via reinforcement learning against these benchmarks discourages probabilistic exploration. Here, we conducted systematic experiments requesting LLMs to produce outputs following simple probabilistic distributions, and found that all modern LLMs tested grossly fail to follow the distributions. For example, requesting a binary output of "1" 49% of the time produces an answer of "0" nearly 100% of the time. This step function-like behavior of near-exclusively generating the output with marginally highest probability even overrules even strong in-built LLM biases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。