用AI生成学术搜索结果摘要,提升信息筛选效率。
AI Overviews in Academic Search: Evaluating AI-generated Summaries of Search Results in a Domain-specific Search Engine
- 在社科领域搜索界面中测试大模型生成的摘要功能。
- 用户使用摘要后主观负担减轻,决策更自信,点击量略降。
- 适合需要快速筛论文的研究者,需注意错误类型与部署风险。
评估学术搜索结果的相关性是一项高负荷任务。本研究通过混合方法设计,考察在社会科学信息检索系统中,以AI生成的搜索结果页面(SERP)级摘要作为辅助功能的效果。首先,人工评估了10个查询的前5个结果摘要,使用两个通用大模型(一个商用、一个开源),归纳出六类典型错误和五项适用于学术场景的部署保障措施。随后开展30人参与的被试内实验,对比有无摘要的界面。验证性分析显示,有摘要时在主观工作量、有用性感知、满意度及决策信心上呈现一致但非显著的优势;探索性分析表明心理负担更低,挫败感也呈下降趋势。行为数据显示,用户很少展开摘要,且在摘要可用时略微减少结果点击与查询重构次数。结合信息觅食理论与用户反馈,我们提出AI摘要可能通过聚焦页面信息线索,支持早期筛选。总体而言,结果表明该摘要功能是情境与用户相关的辅助工具,而非普适优化,但为学术搜索提供了错误分类、防护建议及具体设计启示。
原文摘要 · Abstract (English)
Evaluating search engine results pages (SERPs) to assess result relevance is a demanding step in academic search. In a formative mixed-methods design study, we examine AI-generated SERP-level summaries as a support feature in an academic search engine for social science information. First, we manually evaluated summaries of the top five results for 10 queries using two general-purpose models, one commercial and one open, deriving an exploratory six-category error taxonomy and five safeguards for scholarly deployment. We then conducted a within-subjects user study (n = 30) comparing interfaces with and without AI summaries. Confirmatory analyses showed consistent but non-significant trends favoring AI summaries for subjective workload, perceived usefulness, satisfaction, and decision-making confidence. Exploratory analyses suggested lower mental demand, with frustration also tending to be lower. Behaviorally, participants rarely expanded the summaries and descriptively made slightly fewer result clicks and query reformulations when summaries were available. Drawing on Information Foraging Theory and participant feedback, we suggest that AI summaries may concentrate SERP-level information scent to support early triage. Overall, the findings indicate that SERP-level AI summaries are a context- and user-dependent aid rather than a universal improvement, while contributing an error taxonomy, safeguard-aware deployment guidance, and concrete design implications for scholarly search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。