用自动化方法优化大模型的代码调度系统，提升性能并减少资源消耗。
Meta-Harness: End-to-End Optimization of Model Harnesses
- 通过智能代理访问历史代码与运行记录，自动搜索最优调度逻辑。
- 在文本分类中比现有系统高7.7分，上下文令牌减少4倍。
- 在数学推理和编程任务中超越人工设计，适合追求高效部署的开发者。
大型语言模型系统的性能不仅取决于模型权重，还依赖于其调度系统：决定存储、检索和呈现给模型的信息的代码。然而，调度系统仍主要由人工设计，现有文本优化器在此场景下表现不佳，因其过度压缩反馈信息。我们提出Meta-Harness，一种端到端的外层优化系统，用于搜索大模型应用的调度代码。该系统采用智能代理，可访问所有先前候选方案的源代码、评分及执行轨迹。在在线文本分类任务中，Meta-Harness相比先进上下文管理方案提升7.7分，同时上下文令牌使用量减少4倍；在检索增强型数学推理任务中，单个发现的调度系统使200道国际数学奥林匹克级问题的平均准确率提升4.7分，覆盖五种独立测试模型；在代理式编程任务中，发现的调度系统超越最佳人工基线，在TerminalBench-2上表现更优。这些结果表明，对过往经验的丰富访问能实现自动化调度工程。
原文摘要 · Abstract (English)
The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On agentic coding, discovered harnesses surpass the best hand-engineered baselines on TerminalBench-2. Together, these results show that richer access to prior experience can enable automated harness engineering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。