将多模态指令转为纯文本，实现手机端隐私保护的AI交互。
ReVision: A Dataset and Baseline VLM for Privacy-Preserving Task-Oriented Visual Instruction Rewriting
- 用轻量级模型把图像+文字指令转成纯文字命令。
- 39,000条数据训练，量化后模型小于500MB仍有效。
- 适合开发不传图、保隐私的本地化智能应用。
随着增强现实、虚拟现实及带强大摄像头的智能手机成为人机交互主要界面，高效且保护隐私的多模态交互变得至关重要。现有强大的视觉语言模型（VLM）通常依赖云端处理，引发两大问题：一是敏感视觉数据上传服务器带来的隐私风险，二是难以实现实时、本地运行。本文提出视觉指令重写（Visual Instruction Rewriting）新方法，将多模态指令转化为纯文本命令，使轻量级本地化指令重写VLM（250M参数）可无缝接入现有对话式AI系统，提升视觉数据隐私保护。为此，我们构建了一个涵盖14个领域的超39,000条样本的数据集，并开发了一款紧凑型VLM——在图像描述数据集上预训练，再针对指令重写任务微调。实验通过NLG指标（如BLEU、METEOR、ROUGE）和语义解析分析验证，即使经过量化（存储占用<500MB），模型仍能有效完成指令重写，支持面向隐私保护的多模态人工智能应用落地。
原文摘要 · Abstract (English)
Efficient and privacy-preserving multimodal interaction is essential as AR, VR, and modern smartphones with powerful cameras become primary interfaces for human-computer communication. Existing powerful large vision-language models (VLMs) enabling multimodal interaction often rely on cloud-based processing, raising significant concerns about (1) visual privacy by transmitting sensitive vision data to servers, and (2) their limited real-time, on-device usability. This paper explores Visual Instruction Rewriting, a novel approach that transforms multimodal instructions into text-only commands, allowing seamless integration of lightweight on-device instruction rewriter VLMs (250M parameters) with existing conversational AI systems, enhancing vision data privacy. To achieve this, we present a dataset of over 39,000 examples across 14 domains and develop a compact VLM, pretrained on image captioning datasets and fine-tuned for instruction rewriting. Experimental results, evaluated through NLG metrics such as BLEU, METEOR, and ROUGE, along with semantic parsing analysis, demonstrate that even a quantized version of the model (<500MB storage footprint) can achieve effective instruction rewriting, thus enabling privacy-focused, multimodal AI applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。