用大模型从代码反推用户故事，小模型加示例效果胜过大模型
Reverse Engineering User Stories from Code using Large Language Models
- 用五种大模型+六种提示策略，从1750个C++代码片段中恢复用户故事
- 200行以内的代码平均F1达0.8，小模型（8B）加示例可媲美大模型（70B）
- 单个示例比复杂推理更有效，适合修复老旧系统文档的开发者
用户故事在敏捷开发中至关重要，但在遗留系统和文档缺失的项目中常被忽略或过时。本文研究大语言模型（LLMs）能否直接从源代码自动恢复用户故事，以及提示设计对输出质量的影响。基于1,750个不同复杂度的标注C++代码片段，评估了五种前沿大模型在六种提示策略下的表现。结果表明，所有模型在200 NLOC以内的代码上平均F1得分达到0.8。关键发现是：仅使用一个示例提示，80亿参数的小模型即可达到700亿参数大模型的性能；而通过链式思维（Chain-of-Thought）进行结构化推理，仅对大模型带来微弱提升。
原文摘要 · Abstract (English)
User stories are essential in agile development, yet often missing or outdated in legacy and poorly documented systems. We investigate whether large language models (LLMs) can automatically recover user stories directly from source code and how prompt design impacts output quality. Using 1,750 annotated C++ snippets of varying complexity, we evaluate five state-of-the-art LLMs across six prompting strategies. Results show that all models achieve, on average, an F1 score of 0.8 for code up to 200 NLOC. Our findings show that a single illustrative example enables the smallest model (8B) to match the performance of a much larger 70B model. In contrast, structured reasoning via Chain-of-Thought offers only marginal gains, primarily for larger models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。