发现自动评价指标在真实性与区分度间存在根本矛盾。
RECOM: A Validity Discrimination Tradeoff in Automatic Metrics for Open Ended Reddit Question Answering
- 构建1.5万条真实社区回复数据集，验证指标有效性与区分能力。
- 现有指标要么能区分真假答案，要么能排序模型优劣，无法兼顾。
- 建议同时报告双轴指标并设定随机基线，避免误判。
自动评估指标常被用于衡量大模型生成文本的质量，但其需同时完成两项任务：区分真实内容与表面相似（真实性），以及区分不同系统性能（区分力）。在开放式、观点驱动的问答场景中，这两者存在冲突。本文提出RECOM（Reddit Evaluation for Correspondence of Models），一个无污染的数据集，包含15,000条r/AskReddit问题及其在模型训练截止日期后产生的真实社区回复。对五个7–10B参数量的开源大模型进行评估，每个指标与随机置换噪声基线对比后发现，无一指标能同时胜任两项任务。余弦相似度可有效区分真实与随机回答（Cohen's d ≈ 2），但无法区分模型表现（|d| < 0.1）；BERTScore精度看似能区分模型（原始|d|达0.63），但在控制回复长度后坍缩至|d| = 0.09，且真实性表现弱（d ≈ 0.8，低于余弦的≈2）。由于所有指标对相同输出评分，该权衡是指标本身的属性，源于表示设计。三名独立人类评估者复现了真实性差距，但同样难以区分模型。我们建议在报告中同时呈现两个维度，并明确设置随机基线。
原文摘要 · Abstract (English)
Automatic metrics are the default for evaluating LLM-generated text, yet a metric is quietly asked to do two jobs: tell genuine content alignment from surface coincidence (validity), and tell a better system from a worse one (discriminative power). On open-ended, opinion-driven question answering, the two are in tension. We introduce RECOM (Reddit Evaluation for Correspondence of Models), a contamination-free evaluation dataset of 15,000 r/AskReddit questions (September 2025), each paired with its authentic community replies, which postdate every evaluated model's training cutoff. Scoring five open-source LLMs (7--10B) against every reply each metric paired with a random-derangement noise floor we find that no metric does both jobs well. Cosine similarity separates real from random answers (Cohen's $d \approx 2$) but cannot rank the five models ($|d| < 0.1$); BERTScore precision appears to rank the models (raw $|d|$ up to 0.63), but once response length is controlled this collapses to $|d| = 0.09$ and its validity is weak ($d \approx 0.8$, versus cosine's $\approx 2$). Because every metric scores the same outputs, this validity--discrimination tradeoff is a property of the metrics, not the models, and we argue it stems from representation design. Three independent LLM judges reproduce the validity gap and likewise separate the five models only weakly. We recommend reporting metrics on both axes, with an explicit random-baseline floor. RECOM is publicly available at https://anonymous.4open.science/r/recom-D4B0
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。