测试通过仍出错？揭示LLM应用中四个隐藏缺陷漏洞。
All Green, Still Broken: Real-Flow Verification Lessons from an LLM-Integrated, Multi-Market Web Application
- 识别出四个组件测试无法覆盖的缺陷逃逸边界。
- 44%的修复涉及运行时、多市场、端到端流程等深层问题。
- 提供可落地的团队自查方法，定位最高缺陷风险环节。
现代网页应用融合大语言模型输出、多市场国际化和浏览器前端与外部数据源交互，测试难度极高。我们研究了一个生产环境中的租赁搜索助手，其自动化测试集在六周内增长至1,553个用例，虽持续通过，用户缺陷仍不断进入生产环境。分析项目中252次缺陷修复提交，按逃逸边界分类发现：约44%的修复涉及四个组件级单元测试无法观察的环节——实时浏览器运行时、非默认市场、端到端流程、全系统层面。一次未在关键边界设防护的修复导致缺陷两次上线。本文提出四边界框架，揭示缺陷分布，并分享实践方法，包括团队可快速定位高修复密度边界的简易策略。
原文摘要 · Abstract (English)
Modern web applications increasingly combine three ingredients that are hard to test: output from large language models, multi-market internationalization, and browser-driven front-ends over external data sources. We report on a production rental-search assistant whose automated suite grew to 1,553 test cases in six weeks. The suite passed continuously, yet user-facing defects continued to reach production. We studied all 252 bug-fix commits in the project and classified each by the boundary, or seam, it escaped through. About 44 percent of the fixes fall in four seams that component-level unit tests cannot observe: the live browser runtime, the non-default market, the end-to-end flow, and the whole-system level. A fix without a guard at the seam let one defect ship twice. We present the four-seam framework, the measured defect distribution, and the practices we adopted, including a simple way for a team to find the seam that carries the most fixes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。