研究大模型处理多文档时性能下降原因，发现实例数量影响比上下文长度更大。
Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Length
- 测试大模型在多实例任务中的表现，发现性能随实例数先微降后崩溃。
- 小规模实例（20-100个）性能轻微下降，大规模时出现严重崩溃。
- 适合关注大模型多任务处理能力优化的研究者和应用开发者。
用户常依赖大语言模型（LLMs）处理多个文档或进行多实例分析。例如，分析多条电影评论的情感需逐条处理后聚合结果。尽管单个任务性能优异，但对多实例输入的处理能力研究仍不足。本文全面评估了大模型在单任务表现优异的多实例处理（MIP）能力。结果显示，所有大模型均呈现先轻微性能下降（约20-100个实例），随后在实例数量增加时发生性能崩溃。分析表明，虽然上下文长度相关，但实例数量对最终结果的影响更强。因此，在优化大模型多实例处理性能时，应同时关注上下文长度与实例数量，尤其重视实例数量的影响。
原文摘要 · Abstract (English)
Users often rely on Large Language Models (LLMs) for processing multiple documents or performing analysis over a number of instances. For example, analysing the overall sentiment of a number of movie reviews requires an LLM to process the sentiment of each review individually in order to provide a final aggregated answer. While LLM performance on such individual tasks is generally high, there has been little research on how LLMs perform when dealing with multi-instance inputs. In this paper, we perform a comprehensive evaluation of the multi-instance processing (MIP) ability of LLMs for tasks in which they excel individually. The results show that all LLMs follow a pattern of slight performance degradation for small numbers of instances (approximately 20-100), followed by a performance collapse on larger instance counts. Crucially, our analysis shows that while context length is associated with this degradation, the number of instances has a stronger effect on the final results. This finding suggests that when optimising LLM performance for MIP, attention should be paid to both context length and, in particular, instance count.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。