TADDLE可精准识别大模型生成审稿意见中的六类缺陷。
TADDLE: A Tool-Augmented Agent for Detecting Deficient LLM-Generated Peer Reviews
- 用四个专用工具分解审稿缺陷检测任务，由智能体协同调度。
- 在1800篇ICLR 2025论文审稿上实现高精度多标签分类。
- 首个专家标注的缺陷数据集，适合评测和提升审稿质量系统。
大模型生成的同行评审日益普遍，但其缺陷难以察觉，因内容通常流畅且结构规范。现有方法或仅判断作者身份，或使用针对人工撰写的特征评分质量；尚无系统能以细粒度缺陷类型检测大模型生成的审稿问题。为此，我们提出TADDLE——一个用于检测大模型生成审稿缺陷的工具增强型智能体，并构建了首个专家标注的基准数据集。该数据集包含对50篇ICLR 2025论文的1,800篇审稿意见，由18位领域专家按六类缺陷类别（外加非缺陷标签）进行多标签标注。TADDLE将检测任务分解为四个专用分析工具：Verify、Correct、Complete与Transform，由智能体协调；集成器通过两阶段半监督学习将输出合成二分类与多标签分类结果。大量实验表明，TADDLE在二分类与多标签分类任务中均表现优异。相关基准与代码已开源：https://github.com/AquariusAQ/TADDLE。
原文摘要 · Abstract (English)
LLM-generated peer reviews are increasingly common at major venues, yet their deficiencies are hard to detect because they are uniformly fluent and well-structured. Existing work either classifies authorship without judging quality, or scores quality with features designed for human-written reviews; no prior system detects deficiencies in LLM-generated reviews at the level of individual defect types. To bridge the gap, we introduce TADDLE, a Tool-Augmented Agent for Detecting Deficient LLM-Generated Peer Reviews, together with the first expert-annotated benchmark for this task. Our benchmark comprises 1,800 reviews on 50 ICLR 2025 papers, multi-label-annotated by 18 domain experts against a taxonomy of six defect categories (plus a non-deficient label). TADDLE decomposes detection into four specialized analysis tools -- Verify, Correct, Complete, and Transform -- orchestrated by an agent; an integrator synthesizes their outputs into binary and multi-label classifications via two-stage semi-supervised learning. Extensive experiments show that TADDLE performs strongly on both binary detection and the multi-label classification task. We release the benchmark and code at https://github.com/AquariusAQ/TADDLE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。