用屏幕行为数据微调模型，显著提升意图预测准确率。
Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark
- 基于屏幕上下文行为数据，对大模型进行监督微调。
- 微调后模型在661条测试数据上语义相似度达0.783，远超零样本基线。
- 模型性能高度依赖架构匹配，小模型难复现大模型效果。
我们在PiSAR（Persona, intent, Screen, Action, Rationale）数据集的661行保留子集上，对三种监督微调模型与前沿零样本基线进行对比评测。该数据集包含12,929条来自应用商店评论、美国趋势调查和OPeRA购物轨迹的屏幕锚定行为推理数据。所有模型使用相同测试集与评分流程。结果发现：前沿零样本模型（Claude Opus 4.7 和 GPT-5.5）的语义相似度分别为0.459和0.482；而微调后的Qwen3-VL-8B-Instruct达到0.783，其中79%的样本达到≥0.7的阈值，远高于零样本基线的1-2%。相反，同一训练数据和方法下，Gemma-4-26B-A4B-IT仅得0.441，与零样本模型同量级。这表明存在模型与训练策略的不匹配问题：高参数量的推理优化模型难以被替代，需更多数据或更强微调方法才能释放潜力。
原文摘要 · Abstract (English)
We benchmark three supervised fine-tuned models against frontier zero-shot baselines on a 661-row held-out slice of PiSAR (Persona, intent, Screen, Action, Rationale), a 12,929-tuple corpus of screen-anchored behavioural rationales curated from public app-store reviews, Pew American Trends Panel demographics, and the OPeRA shopper traces. Every model, frontier or fine-tuned, is evaluated on the same 661-row slice with the same scoring pipeline. Two findings. First, frontier zero-shot baselines (Claude Opus 4.7 and GPT-5.5) reach sem_sim 0.459 and 0.482 respectively; a fine-tuned Qwen3-VL-8B-Instruct reaches 0.783 and clears sem_sim >= 0.7 on 79% of rows, against 1-2% for either frontier baseline, a gap of 0.30 absolute on the same test set. Second, the same training data and recipe on Gemma-4-26B-A4B-IT scores only 0.441, in the same band as the frontier zero-shot baselines rather than the fine-tuned Qwen. We read this as a recipe-vs-model mismatch: the reasoning-tuned high-parameter model resists displacement and would likely need either more data or a stronger fine-tuning method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。