用提示工程让AI像专家一样审论文,自动发现方法漏洞。
AI-Driven Scholarly Peer Review via Persistent Workflow Prompting, Meta-Prompting, and Meta-Reasoning
- 设计持续工作流提示,让AI按步骤系统化分析论文
- 能识别实验缺陷、区分论点与证据、验证数据合理性
- 适合科研人员、期刊编辑快速辅助审稿
科学论文的严格同行评审对大型语言模型(LLMs)构成重大挑战,部分源于数据限制和专家推理的复杂性。本文提出持久工作流提示(PWP),一种无需编码、不依赖API的提示工程方法,通过标准LLM对话界面实现广泛适用。我们以实验化学论文为例,构建了基于Markdown结构化的分层模块化提示,采用元提示和元推理技术迭代优化,系统化编码专家评审流程与隐性知识。该提示一次性加载后,可持久触发后续查询,引导现代推理型LLM完成多模态评估。演示显示,该方法在测试案例中成功识别出关键方法学缺陷,缓解输入偏差,完成区分主张与证据、整合文本/图像/图表推断参数、执行定量可行性检查、对比估算与主张、评估先验合理性等复杂任务。为保障透明性与可复现性,补充材料包含完整提示、详细分析及交互日志。本研究不仅适用于特定场景,更揭示了工作流形式化对提升模型复杂科学分析能力的潜力。
原文摘要 · Abstract (English)
Critical peer review of scientific manuscripts presents a significant challenge for Large Language Models (LLMs), partly due to data limitations and the complexity of expert reasoning. This report introduces Persistent Workflow Prompting (PWP), a potentially broadly applicable prompt engineering methodology designed to bridge this gap using standard LLM chat interfaces (zero-code, no APIs). We present a proof-of-concept PWP prompt for the critical analysis of experimental chemistry manuscripts, featuring a hierarchical, modular architecture (structured via Markdown) that defines detailed analysis workflows. We develop this PWP prompt through iterative application of meta-prompting techniques and meta-reasoning aimed at systematically codifying expert review workflows, including tacit knowledge. Submitted once at the start of a session, this PWP prompt equips the LLM with persistent workflows triggered by subsequent queries, guiding modern reasoning LLMs through systematic, multimodal evaluations. Demonstrations show the PWP-guided LLM identifying major methodological flaws in a test case while mitigating LLM input bias and performing complex tasks, including distinguishing claims from evidence, integrating text/photo/figure analysis to infer parameters, executing quantitative feasibility checks, comparing estimates against claims, and assessing a priori plausibility. To ensure transparency and facilitate replication, we provide full prompts, detailed demonstration analyses, and logs of interactive chats as supplementary resources. Beyond the specific application, this work offers insights into the meta-development process itself, highlighting the potential of PWP, informed by detailed workflow formalization, to enable sophisticated analysis using readily available LLMs for complex scientific tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。