用数据增强大模型生成研究想法,提升可行性与质量。
Augmenting Research Ideation with Data: An Empirical Investigation in Social Science
- 生成阶段引入元数据,引导模型产出更可行的想法。
- 自动预验证步骤使选中想法整体质量提升7%。
- 实证显示辅助后生成的想法更具启发性,适合科研人员使用。
大型语言模型在生成新研究想法方面展现出巨大潜力,但这些想法常缺乏可行性和有效性。本文探究在构思过程中引入相关数据是否能提升想法质量。我们的框架在两个阶段融合数据:(1) 在生成阶段加入元数据,引导模型关注更可行的概念;(2) 在筛选阶段引入自动化预验证,评估假设的实证合理性。我们在社会科学领域,特别是气候谈判主题上评估该方法。专家评估显示,元数据使想法可行性提升20%,自动化验证使最终选中想法的整体质量提升7%。此外,通过人类实验发现,经数据和预验证支持的模型生成想法能有效激发研究人员灵感,参与者在辅助下提出的想法质量显著高于无帮助时。结果表明,数据增强的研究构想具有实际应用价值,对真实学术场景中的大模型辅助研究有重要意义。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) demonstrate strong potential for generating novel research ideas, yet such ideas often struggle with feasibility and effectiveness. In this paper, we investigate whether augmenting LLMs with relevant data during the ideation process can improve idea quality. Our framework integrates data at two stages: (1) incorporating metadata during idea generation to guide models toward more feasible concepts, and (2) introducing an automated preliminary validation step during idea selection to assess the empirical plausibility of hypotheses within ideas. We evaluate our approach in the social science domain, with a specific focus on climate negotiation topics. Expert evaluation shows that metadata improves the feasibility of generated ideas by 20%, while automated validation improves the overall quality of selected ideas by 7%. Beyond assessing the quality of LLM-generated ideas, we conduct a human study to examine whether these ideas, augmented with related data and preliminary validation, can inspire researchers in their own ideation. Participants report that the LLM-generated ideas and validation are highly useful, and the ideas they propose with such support are proven to be of higher quality than those proposed without assistance. Our findings highlight the potential of data-augmented research ideation and underscore the practical value of LLM-assisted ideation in real-world academic settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。