测试大模型能否双向理解代码，发现其推理一致性仍不足。
Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via Invertibility
- 设计四类代码任务，通过无执行的精确匹配评估正反向转换一致性。
- 所有模型在零样本、微调和自省下均表现有限，无法完全修复错误。
- 揭示模型在简单压缩算法上也失败，说明问题不在复杂度而在于内部一致。
大语言模型在代码基准测试中表现良好，但正向与反向执行间的稳定推理仍难以实现。我们提出RoundTripCodeEval（RTCE），一个包含四类代码执行推理任务的基准，通过无需执行的精确匹配评估四种无损压缩算法的双射保真度。在零样本提示、基于执行轨迹的监督微调以及迭代自省三种策略下评估当前最先进的Code-LLMs。所有方法仅带来有限提升，且均未能弥合差距，表明现有模型缺乏可靠双向代码推理所需的内部连贯性。RTCE揭示了现有基准无法捕捉的问题：模型虽能单独通过正向与反向任务，但在组合式往返测试中失败，暴露其内部表示存在互不一致；微调与自省在一轮修订后即饱和，说明无法修复根本性的算法误解；即使在简单双射如RLE上失败也持续存在，表明算法复杂度并非唯一根源。
原文摘要 · Abstract (English)
LLMs demonstrate strong performance on code benchmarks, yet consistent reasoning across forward and backward execution remains elusive. We present RoundTripCodeEval (RTCE), a benchmark of four code execution reasoning tasks that evaluates round-trip consistency through execution-free, exact-match assessment of bijection fidelity across four lossless compression algorithms. We evaluate state-of-the-art Code-LLMs under zero-shot prompting, supervised fine-tuning on execution traces, and iterative self-reflection. All approaches yield only modest improvements and none closes the gap, revealing that current LLMs lack the internal coherence required for reliable bidirectional code reasoning. RTCE surfaces findings invisible to existing benchmarks: models frequently pass individual forward and backward tasks yet fail the combined round-trip, exposing mutually inconsistent internal representations; SFT and self-reflection saturate after one revision round, indicating they cannot repair fundamental algorithmic misunderstandings; and failures persist even on simple bijections such as RLE, suggesting that algorithmic complexity is not the sole root cause.\footnote{Code and dataset are available at https://github.com/Nickil21/round-trip-code-compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。