给大模型'用药'会严重破坏其答题能力，连提示词都成'消耗品'
LLMs on Drugs: Language Models Are Few-Shot Consumers
- 用四种'药物提示'干扰GPT-5-mini推理，测试其可靠性
- 酒精使准确率从45%暴跌至10%，其他药物也显著下降
- 提示词本身就能像'毒品'一样影响模型表现，适合关注安全性的研究者
大型语言模型在推理时对设定的人格角色敏感，但此前从未对提示层面的‘药物’干预进行过严格评估。本研究首次通过受控实验，使用ARC-Challenge数据集对GPT-5-mini进行测试。四组单句提示（LSD、可卡因、酒精、大麻）与清醒对照组相比，每组100个验证项，采用确定性解码、完整日志记录、威尔逊置信区间及费舍尔精确检验。对照组准确率为0.45；酒精组降至0.10（p = 3.2e-8），可卡因组为0.21（p = 4.9e-4），LSD组为0.19（p = 1.3e-4），大麻组为0.30（p = 0.041）。结果主要源于人格提示破坏了强制的“Answer: <LETTER>”格式。因此，提示文本可被视为一种‘少样本消耗品’，无需修改模型权重即可大幅降低其可靠性。所有实验代码、原始结果与分析脚本已公开于https://github.com/lexdoudkin/llms-on-drugs。
原文摘要 · Abstract (English)
Large language models (LLMs) are sensitive to the personas imposed on them at inference time, yet prompt-level "drug" interventions have never been benchmarked rigorously. We present the first controlled study of psychoactive framings on GPT-5-mini using ARC-Challenge. Four single-sentence prompts -- LSD, cocaine, alcohol, and cannabis -- are compared against a sober control across 100 validation items per condition, with deterministic decoding, full logging, Wilson confidence intervals, and Fisher exact tests. Control accuracy is 0.45; alcohol collapses to 0.10 (p = 3.2e-8), cocaine to 0.21 (p = 4.9e-4), LSD to 0.19 (p = 1.3e-4), and cannabis to 0.30 (p = 0.041), largely because persona prompts disrupt the mandated "Answer: <LETTER>" template. Persona text therefore behaves like a "few-shot consumable" that can destroy reliability without touching model weights. All experimental code, raw results, and analysis scripts are available at https://github.com/lexdoudkin/llms-on-drugs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。