只需几个简单词，就能让大模型裁判系统判错，揭示其脆弱性。
AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens
- 用模型自身生成的低困惑度词，可轻易翻转裁判结论。
- 在24个测试场景中，90%以上触发错误判定，远超旧基准。
- 发现的对抗词池可用于防御，提升模型鲁棒性，适合安全研究者。
LLM-as-a-Judge 系统为现代 RLHF 与 RLVR 流水线提供奖励信号，但其二元判决仅依赖隐藏状态中的单一线性读出量 F_gap。我们发现该读出机制过于浅显，仅需短且低困惑度的词即可将判决从“否”翻转为“是”。这些词从裁判模型自身在响应位置的下一个词分布中采样，无需人工设定种子或梯度优化。我们的方法 AdvJudge-Zero 在六种 Qwen、Llama 与 Gemma 裁判模型的 24 个（模型，数据集）组合中，于 22 个场景实现超过 90% 的集成误报率，显著高于先前 10 词基准的 54–72%。所发现的对抗词池可跨格式迁移至 70B 规模的标量奖励模型。同一词池还可用于防御：基于 9 类机制分类的 LoRA 微调能增强跨家族泛化能力，而单纯采样则无效，说明机制多样性优于词池规模。在 GRPO 训练下，强化后的裁判模型消除了基线在 MATH 与 GSM8K 上出现的奖励坍缩问题（误报激增与长度坍缩），在每条件十次随机种子下均稳定表现。所发现的词池、机制分类体系及每提示翻转记录将按负责任披露原则公开。
原文摘要 · Abstract (English)
LLM-as-a-Judge systems supply the reward signal in modern RLHF and RLVR pipelines, but their binary verdict reduces to a single linear readout F_gap on one hidden state. We show this readout is shallow enough that short, low-perplexity tokens flip the verdict from "No" to "Yes". These tokens are sampled from the judge's own next-token distribution at the response position, with no manual seed set and no gradient-based optimization. Our procedure, AdvJudge-Zero, reaches $>$90% ensemble false-positive rate on 22 of 24 (model, dataset) cells across six Qwen, Llama, and Gemma judges, versus 54-72% for the prior curated 10-token benchmark, and the discovered surface transfers cross-format to a 70B scalar reward model. The same discovered pool enables a defense: a LoRA fine-tune stratified by a 9-class mechanism taxonomy hardens cross-family generalization where naive sampling on the same pool fails, with mechanism breadth rather than pool size carrying the gain. Under GRPO training, the hardened judge eliminates the reward-collapse failures (false-positive spikes and length collapse) we observe in the unhardened baseline on both MATH and GSM8K at ten seeds per condition. The discovered pool, the mechanism taxonomy, and per-prompt flip records will be released under responsible disclosure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。