让智能体同时搞定验证码和通用界面操作，还能自我纠错。
CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training
- 构建动态验证码系统，覆盖7类常见类型，测试智能体能力。
- 自动生成带推理过程的交互轨迹，失败案例也用于训练纠错。
- 在真实和合成数据上表现优于基线，兼顾验证码与通用任务性能。
GUI智能体正从多模块流水线转向端到端的原生视觉语言模型（VLMs），直接感知原始屏幕截图并操作设备。尽管通用界面任务进展迅速，验证码破解仍是重大挑战。现有专用验证码系统无法处理通用界面任务。为此，我们提出ReCAP：一个具备验证码求解能力的原生GUI智能体，可在解决现代交互式验证码的同时保持通用任务性能。首先，我们设计了一个涵盖七类代表性验证码的动态系统，以检验基础与互补能力。其次，开发自动化数据收集与清洗流程，生成大规模带有推理痕迹的验证码交互轨迹。由于验证码求解常需多步操作且需从中间错误中恢复，我们进一步利用失败轨迹构建自纠正数据，训练智能体在线反思并修正动作。在合成与真实世界测试集上，ReCAP显著提升验证码求解成功率，同时在通用GUI基准测试中保持优异性能。
原文摘要 · Abstract (English)
GUI agents are rapidly shifting from multi-module pipelines to end-to-end, native vision-language models (VLMs) that perceive raw screenshots and directly interact with digital devices. Despite rapid progress on general GUI tasks, CAPTCHA solving remains a major challenge. On the other hand, although specialized CAPTCHA solving pipelines exist, they cannot handle general GUI tasks. To address this gap, we introduce ReCAP: a CAPTCHA-capable native GUI agent that solves modern, interactive CAPTCHA challenges while retaining general GUI-agent performance. We first develop a dynamic CAPTCHA system spanning seven representative CAPTCHA types, designed to stress primitive and complementary capabilities for CAPTCHA solving. Then, we develop an automated data collection and curation pipeline that generates large-scale CAPTCHA interaction trajectories paired with reasoning traces. As CAPTCHA solving often requires multi-step interaction and recovery from intermediate mistakes, we further leverage failed trajectories to construct self-correction data, training agents to reflect on errors and correct their actions online. Across synthetic and real-world test sets, ReCAP substantially improves CAPTCHA-solving success over its base agents, while maintaining strong performance on general GUI-agent benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。