研究如何高效组织大模型处理结构化数据的上下文，发现模型能力决定成败。
Structured Context Engineering for File-Native Agentic Systems: Evaluating Schema Accuracy, Format Effectiveness, and Multi-File Navigation at Scale
- 用SQL生成任务测试不同格式与架构，覆盖11个模型和万级表规模
- 前沿模型用文件式上下文提升2.7%准确率，开源模型反而下降7.7%
- 模型能力差距达21个百分点，远超格式或架构影响，适合工程部署参考
大型语言模型代理越来越多地通过程序接口操作外部系统，但实践中缺乏关于如何组织代理所消耗上下文的实证指导。本文以SQL生成作为程序化代理操作的代理，对结构化数据的上下文工程进行了系统研究，涵盖9,649次实验，涉及11个模型、4种格式（YAML、Markdown、JSON、Token-Oriented Object Notation [TOON]），以及从10到10,000张表的模式。研究结果挑战了常见假设：首先，架构选择依赖模型性能——文件式上下文检索对前沿模型（Claude、GPT、Gemini）提升2.7%准确率（p=0.029），但对开源模型整体下降7.7%（p<0.001），且表现差异显著；其次，格式对整体准确率无显著影响（卡方=2.45，p=0.484），但部分模型尤其是开源模型存在格式敏感性；第三，模型能力是主导因素，前沿与开源模型间准确率差距达21个百分点，远超格式或架构的影响；第四，通过领域分区模式，文件原生代理可扩展至10,000张表并保持高导航准确率；第五，文件大小不能预测运行效率：紧凑或新颖格式可能因grep输出密度和模式陌生度引入令牌开销，其程度取决于模型能力。研究为在结构化系统上部署LLM代理提供了基于证据的指导，表明架构决策应根据模型能力定制，而非假设通用最佳实践。
原文摘要 · Abstract (English)
Large Language Model agents increasingly operate external systems through programmatic interfaces, yet practitioners lack empirical guidance on how to structure the context these agents consume. Using SQL generation as a proxy for programmatic agent operations, we present a systematic study of context engineering for structured data, comprising 9,649 experiments across 11 models, 4 formats (YAML, Markdown, JSON, Token-Oriented Object Notation [TOON]), and schemas ranging from 10 to 10,000 tables. Our findings challenge common assumptions. First, architecture choice is model-dependent: file-based context retrieval improves accuracy for frontier-tier models (Claude, GPT, Gemini; +2.7%, p=0.029) but shows mixed results for open source models (aggregate -7.7%, p<0.001), with deficits varying substantially by model. Second, format does not significantly affect aggregate accuracy (chi-squared=2.45, p=0.484), though individual models, particularly open source, exhibit format-specific sensitivities. Third, model capability is the dominant factor, with a 21 percentage point accuracy gap between frontier and open source tiers that dwarfs any format or architecture effect. Fourth, file-native agents scale to 10,000 tables through domain-partitioned schemas while maintaining high navigation accuracy. Fifth, file size does not predict runtime efficiency: compact or novel formats can incur a token overhead driven by grep output density and pattern unfamiliarity, with the magnitude depending on model capability. These findings provide practitioners with evidence-based guidance for deploying LLM agents on structured systems, demonstrating that architectural decisions should be tailored to model capability rather than assuming universal best practices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。