用用户评论指导LLM改进编程答案,提升技术问答平台质量
Human-Aligned Enhancement of Programming Answers with LLMs Guided by User Feedback
- 基于用户评论和问题上下文,构建自动化改进工具AUTOCOMBAT
- 在790条问答数据上,模型生成结果接近人工修改水平
- 84.5%开发者愿使用该工具,适合开发人员与知识平台优化
大型语言模型(LLMs)广泛用于代码生成、优化和文档编写,但其在类似Stack Overflow的问答平台上根据用户反馈改进已有答案的能力仍待探索。约三分之一的用户评论因时间、专业或可见性限制未被回应,导致答案不完整或过时。本研究提出ReSOlve基准,包含790个带评论线程的问答数据,标注了改进相关与一般反馈。评估四种先进LLM后发现DeepSeek在准确率与召回率间平衡最佳。进一步提出AUTOCOMBAT工具,结合评论与问题上下文自动改进答案。相比人工修订参考,AUTOCOMBAT生成结果接近人类水平,保留原意且显著优于基线。58名从业者参与的用户研究表明,84.5%表示愿意采纳或推荐该工具。结果表明,可扩展的反馈驱动答案优化具有提升技术知识平台可靠性的潜力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are widely used to support software developers in tasks such as code generation, optimization, and documentation. However, their ability to improve existing programming answers in a human-like manner remains underexplored. On technical question-and-answer platforms such as Stack Overflow (SO), contributors often revise answers based on user comments that identify errors, inefficiencies, or missing explanations. Yet roughly one-third of this feedback is never addressed due to limited time, expertise, or visibility, leaving many answers incomplete or outdated. This study investigates whether LLMs can enhance programming answers by interpreting and incorporating comment-based feedback. We make four main contributions. First, we introduce ReSOlve, a benchmark consisting of 790 SO answers with associated comment threads, annotated for improvement-related and general feedback. Second, we evaluate four state-of-the-art LLMs on their ability to identify actionable concerns, finding that DeepSeek achieves the best balance between precision and recall. Third, we present AUTOCOMBAT, an LLM-powered tool that improves programming answers by jointly leveraging user comments and question context. Compared to human revised references, AUTOCOMBAT produces near-human quality improvements while preserving the original intent and significantly outperforming the baseline. Finally, a user study with 58 practitioners shows strong practical value, with 84.5 percent indicating they would adopt or recommend the tool. Overall, AUTOCOMBAT demonstrates the potential of scalable, feedback-driven answer refinement to improve the reliability and trustworthiness of technical knowledge platforms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。