用结构化推理替代模糊打分,让AI评辩论更透明可靠
GRASP: Deterministic argument ranking in interaction graphs

- 基于攻击与支持传播的确定性算法,逐层分析论点互动
- 局部判断一致性显著高于整体评分,减少模型间分歧
- 专注论点结构强度而非说服力,适合审计与可解释性需求
大型语言模型越来越多被用作自动评判者来评估论点强度。随着这一角色扩展,其可信度依赖于一致性、透明性以及将论证结构与修辞魅力分离的能力。然而我们发现,主流的‘整体评判’模式——即模型对整场辩论给出单一结论——存在显著的模型间分歧。我们认为这种不稳定性源于将辩论的复杂互动结构压缩为一个不透明的分数。为此,我们提出GRASP(渐进式攻击-支持传播排名),一种通过收敛的攻击-防御传播算子,将稳定的局部互动判断聚合为全局排名的确定性框架。我们证明,在LLM作为评判者的评估中,局部互动判断比整体排名更具可重复性,使GRASP能生成更一致的全局排名。进一步发现,GRASP得分与人类‘说服力’标签无相关性,凸显关键的社会技术差异:GRASP不衡量说服力、真实性或修辞吸引力,而是衡量结构充分性——一种基于显式互动图的防御意识型论点鲁棒性。总体而言,GRASP为整体式LLM评判提供了一种透明且可审计的替代方案。
原文摘要 · Abstract (English)
Large language models are increasingly deployed as automated judges to evaluate the strength of arguments. As this role expands, their legitimacy depends on consistency, transparency, and the ability to separate argumentative structure from rhetorical appeal. However, we show that holistic judging - a common LLM-as-a-Judge practice where a model provides a global verdict on a debate - suffers from substantial inter-model disagreement. We argue that this instability arises from collapsing a debate's complex interaction structure into a single opaque score. To address this, we propose GRASP (Gradual Ranking with Attacks and Support Propagation), a deterministic framework that aggregates stable local interaction judgments into a global ranking via a convergent attack--defense propagation operator. We show that local interaction judgments are more reproducible than holistic rankings in LLM-as-a-Judge evaluations, allowing GRASP to produce more consistent global rankings. We further show that GRASP scores do not correlate with human "convincingness" labels, highlighting a vital sociotechnical distinction: GRASP does not measure persuasion, factuality, or rhetorical appeal, but structural sufficiency - a defense-aware notion of argument robustness over the explicit interaction graph. Overall, GRASP offers a transparent and auditable alternative to holistic LLM judging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。