用断言语义提升测试代码摘要质量，让LLM更懂测试意图
Assertion-Aware Test Code Summarization with Large Language Models
- 基于断言语义设计提示词，替代完整测试方法上下文
- 断言语义提示使摘要质量平均提升2.3%（4.45→4.35）
- 适合需要自动生成测试说明的开发者与自动化测试团队
单元测试常缺乏简洁的摘要来传达测试意图，尤其在自动生成或文档缺失的代码库中。大型语言模型（LLMs）提供了有前景的解决方案，但其效果高度依赖于提示方式。与通用代码摘要不同，测试代码摘要面临独特挑战：测试方法通过断言验证预期行为，而非实现功能。本文构建了一个包含91个真实Java测试用例及其开发人员编写摘要的新基准，并开展受控消融实验，研究测试代码相关组件（如被测方法、断言消息、断言语义）对LLM生成测试摘要性能的影响。我们评估了四种代码LLM（Codex、Codestral、DeepSeek、Qwen-Coder）在七种提示配置下的表现，使用n-gram指标（BLEU、ROUGE-L、METEOR）、语义相似度（BERTScore）和基于LLM的评估。结果表明，使用断言语义提示可使摘要质量平均提升0.10分（2.3%），相比完整被测方法上下文（4.45 vs. 4.35），且所需输入令牌更少。Codex和Qwen-Coder在与人工摘要对齐方面表现最佳，而DeepSeek尽管词汇重叠高，但性能较差。复现代码包已公开于https://doi.org/10.5281/zenodo.17067550。
原文摘要 · Abstract (English)
Unit tests often lack concise summaries that convey test intent, especially in auto-generated or poorly documented codebases. Large Language Models (LLMs) offer a promising solution, but their effectiveness depends heavily on how they are prompted. Unlike generic code summarization, test-code summarization poses distinct challenges because test methods validate expected behavior through assertions rather than implementing functionality. This paper presents a new benchmark of 91 real-world Java test cases paired with developer-written summaries and conducts a controlled ablation study to investigate how test code-related components-such as the method under test (MUT), assertion messages, and assertion semantics-affect the performance of LLM-generated test summaries. We evaluate four code LLMs (Codex, Codestral, DeepSeek, and Qwen-Coder) across seven prompt configurations using n-gram metrics (BLEU, ROUGE-L, METEOR), semantic similarity (BERTScore), and LLM-based evaluation. Results show that prompting with assertion semantics improves summary quality by an average of 0.10 points (2.3%) over full MUT context (4.45 vs. 4.35) while requiring fewer input tokens. Codex and Qwen-Coder achieve the highest alignment with human-written summaries, while DeepSeek underperforms despite high lexical overlap. The replication package is publicly available at https://doi.org/10. 5281/zenodo.17067550
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。