通过分析思维过程而非输出,能更有效区分人类与机器。
Process Matters more than Output for Distinguishing Humans from Machines

- 设计30项认知任务,捕捉行为背后的思维过程特征。
- 过程特征分类准确率达AUC 0.88,优于仅看结果表现。
- 特定任务的思维过程训练可提升机器拟人化,但跨任务迁移有限。
随着大语言模型和自主代理在在线场景中广泛应用,可靠的人机辨别日益重要。现有方法侧重输出是否与人类一致,遵循图灵提出的以输出为智能标准的观点。认知科学则主张评估行为生成过程。为此,本文提出CogCAPTCHA30,一套包含30项认知任务的测试集,旨在在任务表现匹配时仍能揭示诊断性过程特征。实验表明,过程特征相比性能指标提供更强判别信号,在输出匹配条件下仍可稳定区分人类与代理(平均过程特征分类器AUC=0.88)。对比了Claude Sonnet 4.5、GPT-5、Gemini 2.5 Pro等前沿代理,以及基于1070万条人类决策微调的Centaur模型,还有针对Qwen2.5-1.5B-Instruct的两种任务特定微调方法:动作级监督微调(A-SFT)和过程级微调(P-SFT),后者直接优化过程特征。结果显示,大规模人类决策微调可改善代理的类人行为过程,而任务特定的过程监督进一步提升模仿效果。然而,当面对跨任务迁移时,若过程目标无法自然泛化,优势显著减弱。这表明,显式过程监督虽能提升行为拟人化,但依赖于合适且任务特定的过程表示,凸显过程定义是实现机器类人认知的关键瓶颈。
原文摘要 · Abstract (English)
Reliable human-machine discrimination is becoming increasingly important as large language models and autonomous agents are deployed in online settings. Existing approaches evaluate whether a system can produce behavior or responses indistinguishable from those of a human, following the emphasis on outputs as a criterion for intelligence proposed by Alan Turing. Cognitive science offers an alternative perspective: evaluating the process by which behavior is produced. To test whether cognitive processes can reliably distinguish humans from machines, we introduce CogCAPTCHA30, a battery of 30 cognitive tasks designed to elicit diagnostic process-level features even when task performance is matched. Across the battery, process-level features provide stronger discriminative signal than performance metrics alone, reliably distinguishing humans from agents even under output matching (mean process-feature classifier AUC = 0.88). To evaluate agentic process differences, we compare off-the-shelf frontier agents (Claude Sonnet 4.5, GPT-5, Gemini 2.5 Pro), Centaur (a language model fine-tuned on 10.7M human decisions), and two task-specific fine-tuning approaches applied to Qwen2.5-1.5B-Instruct: action-level supervised fine-tuning (A-SFT) and process-level fine-tuning (P-SFT), which directly optimizes process features. Broad fine-tuning on human decisions improves human-like task processes relative to off-the-shelf agents, while task-specific process-level supervision further improves behavioral mimicry. However, this advantage diminishes under cross-task transfer when supervised process targets do not naturally generalize across tasks. Explicit process-level supervision can improve human behavioral mimicry, but only if appropriate task-specific process representations are available, highlighting process specification as a bottleneck for achieving human-like cognitive processes in machines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。