测试大模型对近期开放问题的回答可靠性，发现其语义对齐高但用词差异大。
Assessing LLM Reliability on Temporally Recent Open-Domain Questions
- 构建1.5万条2025年Reddit近期问题数据集，评估模型与社区答案的匹配度。
- 所有模型语义相似度超99%，但词面重合率不足8%，显示高度改写而非复制。
- 小模型反超大模型，证明参数量不决定可靠性，需多维度评估生成质量。
大型语言模型（LLMs）在开放域问答中应用日益广泛，但其对近期信息的人类共识对齐程度仍缺乏研究。本文提出RECOM（Reddit Evaluation for Correspondence of Models）基准数据集，包含15,000条2025年9月的近期Reddit问题及其社区生成的参考答案。我们评估了四种开源模型（Llama3.1-8B、Mistral-7B、Gemma-2-9B、GPT-OSS-20B）的表现，使用词汇度量（BLEU、ROUGE）、语义相似度（BERTScore、MoverScore、余弦相似度）及逻辑推理（NLI）进行分析。核心发现为显著的语义-词汇悖论：所有模型余弦相似度超过99%，但BLEU-1重合率低于8%，差距超过90个百分点，表明模型通过大幅改写保持语义一致。MoverScore（51-53%）反映语义对齐的最优传输成本，处于中间位置。此外，模型规模无法预测性能：70亿参数的Mistral-7B在所有指标上优于200亿参数的GPT-OSS-20B。NLI分析显示矛盾率低于7%，说明模型极少生成与人类共识冲突的内容。研究质疑词汇度量在抽象生成评估中的有效性，主张采用多维框架衡量语义保真度。RECOM数据集公开可获取于https://anonymous.4open.science/r/recom-D4B0。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed for open-domain question answering, yet their alignment with human perspectives on temporally recent information remains underexplored. We introduce RECOM (Reddit Evaluation for Correspondence of Models), a benchmark dataset of 15,000 recent Reddit questions from September 2025 paired with community-derived reference answers. We investigate how four open-source LLMs (Llama3.1-8B, Mistral-7B, Gemma-2-9B, and GPT-OSS-20B) respond to these questions, evaluating alignment using lexical metrics (BLEU, ROUGE), semantic similarity (BERTScore, MoverScore, cosine similarity), and logical inference (NLI). Our central finding is a striking semantic-lexical paradox: all models achieve over 99% cosine similarity with references despite less than 8% BLEU-1 overlap, a 90+ percentage point gap indicating that models preserve meaning through extensive paraphrasing rather than lexical reproduction. MoverScore (51-53%) confirms this pattern, occupying an intermediate position that reflects the optimal transport cost of semantic alignment. Furthermore, model scale does not predict performance: Mistral-7B (7B parameters) outperforms GPT-OSS-20B (20B parameters) across all metrics. NLI analysis reveals that contradiction rates remain below 7%, suggesting models rarely generate content that directly conflicts with human consensus. These findings challenge the reliability of lexical metrics for evaluating abstractive generation and argue for multi-dimensional evaluation frameworks that capture semantic fidelity beyond surface-level text matching. The RECOM dataset is publicly available at https://anonymous.4open.science/r/recom-D4B0
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。