用真实专业论坛数据构建评测基准，检验大模型在冷门领域的真实能力。
LPFQA: A Long-Tail Professional Forum-based Benchmark for LLM Evaluation
- 基于7个领域430个实际问题，构建长尾专业问答集
- 主流大模型在深度领域推理任务上表现显著不足
- 适合评估模型在真实专业场景中的泛化与理解能力
大型语言模型在标准推理与问答评测中表现良好，但这些评测往往无法捕捉其在真实专业场景中处理长尾、高专业性知识的能力。本文提出LPFQA，一个基于真实专业论坛讨论的长尾知识评测基准，覆盖7个学术与工业领域，包含430个基于实际专业知识的精心设计任务。该基准评估专业推理、领域术语理解与上下文解读能力，并采用分层难度结构以确保语义清晰与答案唯一性。对多个主流大模型的实验显示，在需要深度领域推理的任务上存在显著性能差距，暴露出现有评测未发现的局限。总体而言，LPFQA提供了一个真实且具有区分度的评估框架，可补充现有评测并指导未来大模型发展。
原文摘要 · Abstract (English)
Large Language Models (LLMs) perform well on standard reasoning and question-answering benchmarks, yet such evaluations often fail to capture their ability to handle long-tail, expertise-intensive knowledge in real-world professional scenarios. We introduce LPFQA, a long-tail knowledge benchmark derived from authentic professional forum discussions, covering 7 academic and industrial domains with 430 curated tasks grounded in practical expertise. LPFQA evaluates specialized reasoning, domain-specific terminology understanding, and contextual interpretation, and adopts a hierarchical difficulty structure to ensure semantic clarity and uniquely identifiable answers. Experiments on over multiple mainstream LLMs reveal substantial performance gaps, particularly on tasks requiring deep domain reasoning, exposing limitations overlooked by existing benchmarks. Overall, LPFQA provides an authentic and discriminative evaluation framework that complements prior benchmarks and informs future LLM development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。