用提示词自动化协调代码、理论与论文,防止研究软件各部分脱节。
Theory Under Construction: Orchestrating Language Models for Research Software Where the Specification Evolves

- 设计可迭代的提示词控制器,动态选择下一步动作以填补工作区缺失。
- 在46个科研项目中验证,工具在90个测试案例上达到0.768的F1值。
- 通过审计收缩循环提升成果一致性,适合长期科研项目开发者使用。
大型语言模型能生成大量代码和研究文本,但科研软件项目需要代码、数学论证、基准测试与公开声明共同演进。现有方法常出现幻觉累积(主张超出代码或理论支持)和不同部分失步的问题。本文提出Comet-H,一种基于提示词的迭代控制机制,将构思、实现、评估、对齐和论文撰写作为统一工作空间中的耦合步骤。控制器根据当前工作区缺口评分选择提示,保留未完成任务的衰减记录,并在文档更新时重新校验代码与基准。提示选择被建模为小规模上下文相关多臂老虎机问题,使用可解释的线性打分器,无需训练策略。我们在20多个领域构建了46个科研软件仓库,深入分析其中的A3项目——一个完全在闭环内构建的Python静态分析工具,在90个案例的基准上达到F1=0.768,优于最佳基线0.364。约400次提交中,所有成功路径后期均以审计与精简循环为主导。
原文摘要 · Abstract (English)
Large language models can now generate substantial code and draft research text, but research-software projects require more than either artifact alone. The mathematical thesis, executable system, benchmark surface, and public claims must mature together, yet often drift apart. We identify two LM-specific failure modes: hallucination accumulation, in which claims exceed what code or theory supports and unsupported assertions propagate across sessions; and desynchronization, in which code, theory, or the model's own world model fall out of alignment. We propose Comet-H, an iterative prompt automaton that orchestrates ideation, implementation, evaluation, grounding, and paper-writing as coupled coordinates of a single workspace state. At each step, a controller selects the next prompt by scoring it against what the workspace currently lacks, carries unfinished follow-up work forward with a half-life, and re-checks the paper and README against the code and benchmarks whenever documentation changes. We frame prompt selection as a small contextual bandit problem over prompt families, with prompts as arms, workspace deficits as context, and a hand-weighted linear score. This transparent scorer, paired with a fading record of unfinished work, bounds long-horizon follow-ups, requires no learned policy, and makes each prompt choice legible from the workspace. We created a portfolio of 46 research-software repositories across two dozen domains. We study A3 in depth, a Python static-analysis tool built entirely within the loop, which reaches (F1 = 0.768) on a 90-case benchmark, compared with a next-best baseline of 0.364. Across approximately 400 commits, we find that audit-and-contraction passes dominate the later phases of every successful trajectory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。