arXiv:2608.09154cs.CL2026-08

提出新框架,让大模型难靠复制原文作弊。

UNSPECIFIC: General Constraint Synthesis for Breaking Copy-and-Paste Shortcut in LLM Instruction Following

论文配图:UNSPECIFIC: General Constraint Synthesis for Breaking Copy-and-Paste Shortcut in LLM Instruction Following
图 1 · 摘自论文原文
  • 用两篇相似文章共性生成约束,减少直接复制。
  • 只强化易满足的约束,保持自然性和挑战性平衡。
  • 在文章和摘要上双重评估,防止表面应付。

大语言模型在遵循复杂指令时,常需处理多条约束。当前广泛使用的回译方法(back-translation)存在严重漏洞:约束生成模型会直接复制参考文本作为具体约束,导致被评估模型只需复制原文即可通过。为此,我们提出UNSPECIFIC框架:通过两篇相似参考文章的共性生成通用约束,减少复制行为;仅对易被轻易满足的约束进行强化,以平衡难度与自然度;并在生成文章及其摘要上双重评估约束满足情况,惩罚表面化响应。基于新闻、故事和博客领域构建了UNSPECIFIC基准,结果表明,合成约束更具挑战性(如GPT-5 Mini满足率从90%降至78%),更自然(人类评分差距提升30%),且有效抑制复制行为。此外发现大量约束仅在表面满足,未贯穿核心叙事。代码与数据集已开源。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly expected to follow long lists of constraints in complex instructions, and synthesizing instructions from a reference document (i.e., back-translation) is a widely used method to measure/enhance LLMs' ability to follow complex instructions. However, this method introduces a critical loophole: the constraint synthesis model copies text from the reference as a very specific constraint and the evaluated LLM trivially satisfies the constraint by copying its text in the response. To address these issues, we propose UNSPECIFIC, a novel framework that synthesizes constraints common to two similar reference articles to reduce copy-pasting, selectively hardens only trivially satisfied constraints to balance difficulty and naturalness, and evaluates satisfaction on both the generated article and its summary to penalize superficial instruction following. Consequently, we built the UNSPECIFIC benchmark on news, story, and blog domains to analyze the copy-pasting behavior of LLMs. Our results show that our synthesized constraints are not only more challenging (e.g., the satisfaction rate of GPT-5 Mini drops from 90% to 78%) and natural (LLM win-rate gap improves by 30%) from a human perspective but also mitigate the copy-pasting. We also find that a large portion of constraints are satisfied superficially (i.e., not satisfied in the core narrative of the article). The code and datasets are released at https://github.com/JeetDSharma/UNSPECIFIC.

指令遵循大模型评测抗作弊

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。