测试大模型在长代码问答中的鲁棒性，发现其对干扰信息敏感。
Robustness and Reasoning Fidelity of Large Language Models in Long-Context Code Question Answering
- 通过控制变量实验，测试答案格式、干扰项和上下文规模的影响
- 在打乱选项和开放问题中性能下降超30%，存在明显脆弱性
- 适合关注代码推理可靠性的工程师和评估者参考
大语言模型（LLMs）越来越多地用于需要对长代码上下文进行推理的软件工程任务，但其在不同输入条件下的鲁棒性仍不明确。我们通过受控消融实验系统研究了长上下文代码问答任务，测试了答案格式、干扰项和上下文规模的影响。将LongCodeBench Python数据集扩展为包含新的COBOL和Java问答集，并在三种设置下评估主流模型：(i) 打乱的多选选项，(ii) 开放式问题，(iii) 包含相关与对抗性无关信息的'针堆草垛'上下文。结果表明，在打乱多选选项和开放式问题中性能显著下降，且在无关线索存在时表现脆弱。研究揭示了当前长上下文评估的局限性，并为评估现代与遗留系统中的代码推理能力提供了更全面的基准。
原文摘要 · Abstract (English)
Large language models (LLMs) increasingly assist software engineering tasks that require reasoning over long code contexts, yet their robustness under varying input conditions remains unclear. We conduct a systematic study of long-context code question answering using controlled ablations that test sensitivity to answer format, distractors, and context scale. Extending LongCodeBench Python dataset with new COBOL and Java question-answer sets, we evaluate state-of-the-art models under three settings: (i) shuffled multiple-choice options, (ii) open-ended questions and (iii) needle-in-a-haystack contexts containing relevant and adversarially irrelevant information. Results show substantial performance drops in both shuffled multiple-choice options and open-ended questions, and brittle behavior in the presence of irrelevant cues. Our findings highlight limitations of current long-context evaluations and provide a broader benchmark for assessing code reasoning in both legacy and modern systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。