arXiv:2605.04665cs.CL2026-05

大模型对语义相同但表达不同的指令,常答错格式,影响评估可靠性。

Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs

论文配图:Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs
图 1 · 摘自论文原文
  • 设计多种改写指令测试模型输出格式稳定性
  • 仅22%的改写指令仍输出正确标签,78%完全偏离答案空间
  • 任务结构比模型本身更决定格式崩溃,需关注输出模式一致性

当请求内容被语义保留地重写时,大语言模型是否仍按原始任务要求的格式回答?我们发现它们经常不能,即使在温度为零的情况下。在针对五种紧凑型2025年大模型和四种任务类型的150个查询评估中,观察到一种系统性失败模式:提示变体导致输出格式崩溃——当原提示要求简洁标签或单个选项时,语义等价的改写提示会诱使模型输出对话式段落,所求格式消失,而精确匹配评估流程却无法察觉错误。为此,我们发布PARACONSIST基准,包含900个提示(150个基础查询,每查询5种词法、句法及语义扩展变体),并引入语义一致性评分,分解为答案一致性、Sentence-BERT语义相似度与长度稳定性。在全词匹配下,仅约22%的闭合格式变体响应包含正确标签,约78%完全脱离答案空间。在本研究中,任务结构是崩溃的主要预测因子,而非模型身份;模型差异由答案一致性和长度稳定性共同决定。因此,稳健性审计应将响应模式保持视为与答案准确性同等重要的首要可靠性目标。

原文摘要 · Abstract (English)

When the substantive content of a request is rewritten, do large language models still answer in the format the original task asked for? We find that they often do not, even at temperature zero. On a 150-query evaluation over five compact 2025-era LLMs and four task types, we observe a systematic failure mode we call prompt-variant output-mode collapse: when a closed-form prompt asks for a bare label or a single choice token, content-preserving prompt variants can push the model into conversational prose, the requested format dissolves, and exact-match evaluation pipelines silently misjudge the result. To make this measurable, we release PARACONSIST, a 900-prompt benchmark of 150 base queries with five lexical, syntactic, and semantic-expansion prompt variants each, and a Semantic Consistency Score that decomposes prompt-variant robustness into answer consistency, sentence-BERT semantic similarity, and length stability. Under a whole-word answer-set match, only ~22% of closed-form variant responses preserve the ground-truth label inside their output, while ~78% drift away from the answer space entirely. In our pool, the dominant predictor of collapse is task structure rather than model identity, with model differentiation jointly carried by answer consistency and length stability. Robustness audits should therefore track response-mode preservation as a first-class reliability target alongside answer accuracy.

大模型输出格式鲁棒性评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。