长代码生成时，单一模型难搞定复杂组合，多智能体更高效。
Compositional Hardness of Code in Large Language Models -- A Probabilistic Perspective
- 用生成次数衡量复杂任务求解难度，越长越难。
- 同一上下文求解比分步多智能体慢数倍，长度增加呈指数恶化。
- 适合研究多智能体协作与长程推理的开发者参考。
在大语言模型（LLM）用于复杂分析任务（如代码生成）时，常见做法是在模型上下文窗口内一次性采样完整解决方案。已有研究表明，在上下文中进行子任务分解（思维链）有助于解决问题。本文指出，LLM在同一上下文窗口内处理多个子任务存在内在困难——即上下文组合难度。该难度通过生成复杂度指标量化，即采样至少一个正确解所需的生成次数。我们发现，将组合问题集中于同一上下文求解的生成复杂度，远高于将其分配给多个智能体的方案，且该差距随解长度呈指数增长。理论证明与实证结果均支持这一发现。
原文摘要 · Abstract (English)
A common practice in large language model (LLM) usage for complex analytical tasks such as code generation, is to sample a solution for the entire task within the model's context window. Previous works have shown that subtask decomposition within the model's context (chain of thought), is beneficial for solving such tasks. In this work, we point a limitation of LLMs' ability to perform several sub-tasks within the same context window - an in-context hardness of composition, pointing to an advantage for distributing a decomposed problem in a multi-agent system of LLMs. The hardness of composition is quantified by a generation complexity metric, i.e., the number of LLM generations required to sample at least one correct solution. We find a gap between the generation complexity of solving a compositional problem within the same context relative to distributing it among multiple agents, that increases exponentially with the solution's length. We prove our results theoretically and demonstrate them empirically.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。