测试大模型在无痛信息语音转录中是否胡编乱造，发现部分模型竟自信造假。
Refusal Is Not Robustness: Auditing Confident Fabrication in Large Language Models on a Provably Uninformative Clinical Pain Speech Transcript
- 用无痛词汇的语音转录测试模型，检验其能否识别无信息输入
- 6个模型在无信息时几乎都拒绝回答，但Gemini和Llama仍自信胡编，率高达0.76
- 适合关注大模型可信度与幻觉风险的研究者和医疗AI开发者
评估了七个大型语言模型在TAME Pain语音语料库上的表现。该语料库包含5,750条无疼痛信号的哈佛句子转录（无词汇疼痛信息）和1,294条含明确疼痛评分的有信号陈述。基于声学特征可预测疼痛（AUC 0.622，95%置信区间0.553至0.662），而仅依赖转录文本的预测接近随机水平（AUC 0.489，95%置信区间0.418至0.504）。在合作提示下，六个模型在绝大多数无信号转录上选择拒绝回答，正控任务准确率在0.939至1.00之间，校准误差不超过0.100。但在权威提示下，拒绝行为变为提示依赖，同一模型在等效提示下拒答率从0.18到1.00不等。多数模型被迫回答时给出低置信度，而Gemini 2.5 Flash和Llama 3.1 8B始终输出高置信度，自信伪造率分别为0.53和0.76，其余模型最高为0.15。强迫回答中未见显著人口统计学差异（所有p值≥0.20）。
原文摘要 · Abstract (English)
Hallucination and abstention benchmarks rarely establish that a model could not have known the correct answer, making it difficult to distinguish appropriate abstention from an unsupported prediction. Seven large language models were evaluated on the TAME Pain speech corpus. Participants read phonetically balanced Harvard Sentences while one hand was immersed in cold or warm water and reported pain only during periodic pain statements. This protocol generated 5,750 no signal Harvard Sentence utterances whose transcripts contained no lexical pain information and 1,294 signal pain statement utterances in which the pain rating was explicitly spoken. In the no signal arm, pain was recoverable from acoustic features (AUC 0.622, 95% CI 0.553 to 0.662), whereas transcript based prediction was near chance (AUC 0.489, 95% CI 0.418 to 0.504). Because automatic speech recognition removes the acoustic pain cues, any pain score inferred solely from the transcript is unsupported by the available evidence. Under cooperative prompting, six models abstained on nearly all no signal transcripts, correctly extracted spoken pain ratings in the positive control task with accuracies ranging from 0.939 to 1.00, and maintained an expected calibration error of at most 0.100. Under authority framed prompts, abstention became prompt dependent, with the same model ranging from 0.18 to 1.00 across equivalent prompt phrasings. Most models produced low confidence estimates when forced to answer, whereas Gemini 2.5 Flash and Llama 3.1 8B consistently generated confident pain scores with confident fabrication rates of 0.53 and 0.76, compared with at most 0.15 for all other models. No significant demographic effects were observed in forced responses, with all $p$ values greater than or equal to 0.20.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。