实测发现大模型常偷懒漏答,但忘指令问题没想象严重。
Quantifying Laziness, Decoding Suboptimality, and Context Degradation in Large Language Models
- 通过三组实验量化模型在复杂指令下的偷懒行为
- 多数模型未能完成多部分要求,但解码选择基本正确
- 长对话中模型记性出乎意料好,对指令遗忘不明显
大型语言模型常表现出行为缺陷,如偷懒(提前终止回应或部分响应多段请求)、解码次优性(因短视策略未选更优序列)以及上下文退化(长期对话中遗忘或忽略核心指令)。我们对多个先进模型(OpenAI GPT-4 变体、DeepSeek)进行了三组受控实验(A、B、C),量化这些现象。结果显示:模型在满足复杂多部分指令时普遍存在偷懒问题,常遗漏必要内容或未达长度要求,尽管提示明确。但在简单推理任务中,解码次优性证据有限——模型贪婪选择与高置信度解一致;在200轮混乱对话测试中,模型对关键事实和指令的保持远超预期,表现惊人稳健。这表明,虽然严格遵循详细指令仍是挑战,但现代模型在基础检索场景中已内建机制缓解部分预设失败模式(如上下文遗忘)。我们讨论其对可靠性的影响,关联既有指令遵循与长上下文处理研究,并建议采用自反思与动态提示等策略以减少偷懒、提升多指令遵从性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often exhibit behavioral artifacts such as laziness (premature truncation of responses or partial compliance with multi-part requests), decoding suboptimality (failure to select higher-quality sequences due to myopic decoding), and context degradation (forgetting or ignoring core instructions over long conversations). We conducted three controlled experiments (A, B, and C) to quantify these phenomena across several advanced LLMs (OpenAI GPT-4 variant, DeepSeek). Our results indicate widespread laziness in satisfying complex multi-part instructions: models frequently omitted required sections or failed to meet length requirements despite explicit prompting. However, we found limited evidence of decoding suboptimality in a simple reasoning task (the models' greedy answers appeared to align with their highest-confidence solution), and we observed surprising robustness against context degradation in a 200-turn chaotic conversation test - the models maintained key facts and instructions far better than expected. These findings suggest that while compliance with detailed instructions remains an open challenge, modern LLMs may internally mitigate some hypothesized failure modes (such as context forgetting) in straightforward retrieval scenarios. We discuss implications for reliability, relate our findings to prior work on instruction-following and long-context processing, and recommend strategies (such as self-refinement and dynamic prompting) to reduce laziness and bolster multi-instruction compliance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。