测试大模型创意生成中跨领域种子检索的效果
Read, Grep, and Synthesize: Diagnosing Cross-Domain Seed Exposure for LLM Research Ideation
- 用读取、搜索、脚本提取论文关键机制作为种子
- 跨领域检索比同域或无检索更易产出新颖方法
- 随机多样种子与跨域检索效果相近，说明仍难理解为何选这些
发现新方法是机器学习领域的持续循环，常源于技术在不同领域间的迁移。本文通过PaperGym三阶段流程研究当前大模型创意生成系统是否真正受益于跨领域检索，还是仅依赖多样化机制的暴露。第一阶段在隔离环境内使用工具增强的读取、grep和bash进行种子提取；第二阶段通过跨七个机器学习领域的改写式检索实现跨领域种子获取；第三阶段基于评分标准由人工评委对合成方法进行评估。工具增强提取提升了精准度，改写检索扩大了领域覆盖。在方法合成阶段，跨领域检索相比无检索和同域基线获得了更多新颖性胜出，但与随机多样种子对照组无显著差异。结果表明，大模型创意系统确实受益于多样种子暴露，但尚未能有效利用特定种子被检索的语义原因。项目代码、种子库与运行脚本已开源。
原文摘要 · Abstract (English)
The discovery of novel methodologies for emerging problems is a continuing cycle in ML, often driven by the migration of techniques across domains. Building on this observation, we ask whether current LLM ideation systems benefit from targeted cross-domain retrieval or simply from exposure to diverse mechanisms. We study this question through PaperGym, a three-stage pipeline: (1) tool-augmented seed extraction via read, grep, and bash over an isolated paper environment, (2) cross-domain seed retrieval via paraphrasing across seven ML domains, and (3) method synthesis from retrieved seeds, each scored by rubric-based judges. Tool-augmented extraction improves specificity, and paraphrase-based retrieval broadens domain coverage. In synthesis, cross-domain retrieval receives more pairwise novelty wins than no-retrieval and same-domain baselines, but shows no significant difference from a random diverse-seed control. These findings suggest LLM ideation systems benefit from diverse seed exposure, but do not yet reliably exploit the semantic reason particular seeds were retrieved. We release the seed library, rubric prompts, and run scripts at https://github.com/yunjoochoi/PaperGym
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。