首个高效提取大模型评判能力的框架,支持多评估协议。
JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols

- 利用跨协议一致性,用点对点评分生成配对与列表评分监督信号。
- 在有限查询下实现点对点、配对、列表评估最高73.3%、87.0%、71.6%准确率。
- 适用于不同模型规模与防御策略,适合安全研究与评测系统设计者。
大型语言模型(LLM)评判器被广泛应用于各类评估场景,其判断能力成为重要知识产权。然而,黑盒访问使其易受模型提取攻击。现有方法未专门针对LLM评判器,且在查询预算受限时对多评估协议支持不足。本文提出JUDGESTEALER,首个面向点对点评分、配对比较和列表排序三种协议的高效模型提取框架。该方法利用各协议间强一致性,仅通过点对点输入获取评分并转化为配对与列表监督,无需额外查询。通过基于语义多样性、预测不确定性及潜在偏见动态选择输入,提升信息量与查询效率。结合分数平滑与多协议复审机制,保持评分序关系并缓解代理模型适应中的灾难性遗忘。在主流LLM-as-a-judge与奖励模型上实验表明,JUDGESTEALER显著优于基线,在点对点、配对、列表评估中分别达到73.3%、87.0%、71.6%准确率。该框架对不同代理模型规模、适配策略与推理设置均有效,并具备对典型防御手段的鲁棒性。
原文摘要 · Abstract (English)
Large language model (LLM) judges are increasingly used across various evaluation scenarios, making their judgment capabilities valuable intellectual property. However, black-box access exposes these capabilities to model extraction attacks. Existing extraction methods do not specifically target LLM judges and provide limited support for multiple evaluation protocols under restricted query budgets. In this study, we propose JUDGESTEALER, the first query-efficient model extraction framework for replicating judging capabilities across pointwise scoring, pairwise comparison, and listwise ranking protocols. JUDGESTEALER exploits the strong cross-protocol agreement to acquire pointwise scores and transform them into pairwise and listwise supervisions without additional victim queries. To capture informative judge patterns and improve query efficiency, JUDGESTEALER dynamically selects pointwise inputs based on semantic diversity, predictive uncertainty, and potential judge biases. It further applies score smoothing and multi-protocol review to preserve the ordinal structure of scores and mitigate catastrophic forgetting during surrogate adaptation. Extensive experiments on state-of-the-art LLM-as-a-judge and reward models show that JUDGESTEALER consistently outperforms existing extraction baselines, achieving up to 73.3%, 87.0%, and 71.6% accuracy for pointwise, pairwise, and listwise evaluation, respectively. JUDGESTEALER also remains effective across different sur- rogate model scales, adaptation strategies, and reasoning settings. Moreover, JUDGESTEALER demonstrates robustness against representative extraction defenses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。