语言模型代理对语义改写更敏感，易引发深层错误
AgentDiff: Meaning-Bearing Rewrites Trigger Deeper Divergence than Presentation Changes in LLM Agents
- 通过对比语义重写与格式调整，发现前者导致更大答案不一致
- 语义改写使答案不一致性提升19.69个百分点，显著高于格式变化
- 揭示了'隐性发散'机制，适合关注模型可靠性研究者阅读
大型语言模型代理应基于输入含义作答，而非表现形式。我们发现它们并未同等对待这两种变化。AgentDiff衡量在匹配扰动严重度的前提下，语义重写（如改写和同义替换）与呈现变化（如重新排序、格式调整和干扰项）所引发的答案差异。在覆盖十个LLM、七种架构家族、三个基准测试、1530个原始问题及约11150个变体的68组模型-基准-框架组合中，语义重写导致不一致性率高出19.69个百分点（配对t=9.58，p<0.0001；64/68组为正）。该结果在四种严重度代理下稳定（+18.9至+20.9个百分点，全部p<0.0001），在非Qwen家族的48组中仍保持+11.10个百分点。对qwen2.5-14B-Instruct进行全留出评估，验证了该效应结构：预注册的可处理子集在4个留出组中有3个为正，合并后差异依然显著（Welch t=3.81，p=9.6×10⁻⁴）。轨迹分析显示，语义重写保留首步动作，但从第二步起思想相似度下降5.6–10.5点，且引发的连锁反应延长0.17步（配对t=7.69，p=2.5×10⁻¹⁴），这一模式称为‘隐性发散’。AgentDiff因此建立了一个可复现的方向性鲁棒性差距，识别出最可靠的效应区间，并将最终答案不一致与独特的轨迹级特征关联。代码、扰动数据、轨迹和分析脚本已公开于https://anonymous.4open.science/r/agentdiff-emnlp-0BB4/
原文摘要 · Abstract (English)
LLM agents should respond to what an input means, not how it is presented. We show that they do not treat these two kinds of variation equally. AgentDiff measures the difference between answer changes caused by meaning-bearing rewrites (paraphrase and synonym substitution) and presentation changes (reordering, formatting, and distractors), while matching perturbation severity. Across 68 model--benchmark--scaffold cells spanning ten LLMs from seven architecture families, three benchmarks, 1{,}530 original questions, and approximately 11{,}150 variants, meaning-bearing rewrites produce a $+19.69$ percentage-point higher inconsistency rate (paired $t=9.58$, $p<0.0001$; 64/68 cells positive). The result is stable under four severity proxies ($+18.9$ to $+20.9$~pp, all $p<0.0001$) and remains $+11.10$~pp on the 48 cells outside the qwen family. A fully held-out qwen2.5-14B-Instruct evaluation then tests the structure of this effect on 1{,}800 new trajectories: the pre-registered capable-and-tractable partition is positive in 3/4 held-out cells and remains sharply separated after pooling (Welch $t=3.81$, $p=9.6\!\times\!10^{-4}$). Trace analysis explains how the difference propagates. Meaning-bearing rewrites preserve the first action but reduce thought similarity from step 2 onward by $5.6$--$10.5$ points and extend the resulting cascade by $0.17$ steps (paired $t=7.69$, $p=2.5\!\times\!10^{-14}$), a pattern we call \emph{stealth divergence}. AgentDiff therefore establishes a reproducible directional robustness gap, identifies the regime in which it is most reliable, and connects final-answer inconsistency to a distinct trajectory-level signature. Code, perturbations, trajectories, and analysis scripts are released for review at https://anonymous.4open.science/r/agentdiff-emnlp-0BB4/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。