测试大模型在跨会话编码任务中的协作能力，发现其长程记忆严重不足。
From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions
- 构建合成数据集MemoryCode，模拟跨会话编码任务
- GPT-4o等模型在跨会话指令下性能显著下降
- 核心问题是无法有效追踪和整合长链指令信息
大型语言模型（LLMs）在各类任务中表现优异，尤其擅长独立解决单一问题。但它们能否在长期交互中有效协作？为此，我们引入MemoryCode——一个合成的多会话数据集，用于测试模型在杂乱信息中追踪并执行简单编码指令的能力，模拟真实工作场景。所有测试模型在处理孤立指令时表现良好，但即使是最先进的GPT-4o，在指令分散于多个会话时性能也明显下降。分析表明，这源于模型在长指令链中无法有效检索和整合信息。结果揭示了当前大模型在长期协作中的根本局限。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly used in working environments for a wide range of tasks, excelling at solving individual problems in isolation. However, are they also able to effectively collaborate over long-term interactions? To investigate this, we introduce MemoryCode, a synthetic multi-session dataset designed to test LLMs' ability to track and execute simple coding instructions amid irrelevant information, simulating a realistic setting. While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions. Our analysis suggests this is due to their failure to retrieve and integrate information over long instruction chains. Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。