用大模型测试终端界面,发现随机探索更高效,输入生成是关键突破。
Can LLMs Test Terminal User Interfaces?

- 构建跨语言的无头测试基准,覆盖Rust/Go/Python/TypeScript框架
- 随机探索在时间预算下表现最佳,但大模型更高效发现输入锁死缺陷
- 自动推导启动输入带来最大实用提升,适合作为开发者工具的测试参考
终端用户界面(TUI)结合了图形界面的状态感知与终端部署优势,广泛用于开发工具。我们调研197个真实TUI应用:仅12%的测试代码真正执行界面交互,45%的测试从未发送输入,仅检查静态画面。我们构建了一个无头测试基准,涵盖ratatui/Rust、bubbletea/Go、textual/Python和ink/TypeScript,并以可测容器化形式封装。记录行覆盖率、组件覆盖率、渲染状态及崩溃信息。在相同运行时间内,对比四种前沿大模型与随机探索。结果表明:随机探索是强基线,其崩溃发现优势源于更高吞吐;而大模型在每轮交互中更高效,且能唯一触发输入依赖型故障。自动推导启动输入带来最大实际收益,使原本无法启动的应用得以测试。行覆盖率难以预测崩溃发现,削弱其作为测试有效性代理的作用。自动化TUI测试可行但尚未完善,诚实基线比模型选择更重要。我们开源了覆盖率工具tuicov与测试框架tuibot。
原文摘要 · Abstract (English)
Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment and are now common in developer tools. Yet they lack a dedicated testing methodology. We survey 197 real-world TUI applications: only 12% of test code exercises the interface, and 45% of those tests never send input, checking a static frame instead. We turn these applications into a headless benchmark spanning ratatui/Rust, bubbletea/Go, textual/Python, and ink/TypeScript, packaging each as an instrumented Docker image. We record line and widget coverage where reliable, rendered terminal states, and crashes. Under equal wall-clock budgets, we compare four frontier LLMs with random exploration. No model dominates. Random is a strong time-budgeted baseline, but its crash advantage comes from higher throughput: per interaction, LLM guidance is more efficient and uniquely reaches input-gated faults. Automatically deriving launch inputs yields the largest practical gain, enabling applications that otherwise never start. Line coverage poorly predicts crash discovery, weakening it as a proxy for test effectiveness. Automated TUI testing is feasible but far from solved, and honest baselines matter more than model choice. We release the coverage tool tuicov at https://github.com/tui-testing/tuicov and the testing framework tuibot at https://github.com/tui-testing/tuibot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。