arXiv:2608.15614cs.CVcs.AI2026-08中稿 · ECCV

用软件实现轻量级眼动预测,让手机实时处理智能眼镜视频

EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input

论文配图:EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input
图 1 · 摘自论文原文
  • 设计轻量双流模型,仅15.7M参数,可在手机上实时预测视线焦点
  • 预测视线与真实视线效果相当,在10项测试中无显著差异
  • 无需眼动硬件,即可实现高效视频裁剪,适合移动设备部署

将多模态大模型(MLLM)用于可穿戴设备的自指视频理解时,受限于令牌预算。内存和计算成本随视觉令牌数量增长,高分辨率视频在传输和处理上代价高昂。先前工作(GazeLLM)通过聚焦用户注视区域裁剪视频,使视觉令牌数减少约十倍,同时保持或提升完整描述质量。但该方法依赖专用眼动追踪硬件,无法在消费级智能眼镜上使用。我们提出EgoGazeLite,一种轻量级双过程眼动预测模型,无需眼动硬件即可实现软件替代。在两个MLLM、三个自动评估指标及两名LLM评委下,预测注视区域裁剪结果与真实注视区域无显著差异,10次测试全部等效。模型仅需15.7M参数、6.71 GFLOPs,可在消费级加速器上以21.6毫秒/帧速度完成端到端的眼动与裁剪流程。该成果消除了对眼动硬件的需求,实现了无需硬件支持的高效、注视引导式自指视频理解。

原文摘要 · Abstract (English)

The use of multimodal LLMs (MLLMs) for egocentric video understanding with wearable devices is constrained by the token budget. Memory and compute cost scale with the number of visual tokens, and high-resolution video quickly becomes expensive to transmit and process at scale. Prior work (GazeLLM) addresses this by cropping the video around the camera wearer's gaze. This reduces the number of visual tokens by about tenfold while maintaining or improving the quality of full-resolution descriptions. However, this compression strategy depends on dedicated eye-tracking hardware, which is unavailable on consumer smart glasses. Building a software-only substitute poses a joint constraint: the predictor must be accurate enough to preserve downstream description quality, yet light enough to run on-device, within the power and compute budget of a smartphone. We address this with EgoGazeLite, a lightweight dual-process gaze predictor for egocentric video. Across two MLLMs, three automated metrics, and two LLM judges, predicted-gaze crops show no significant difference from ground-truth-gaze crops. Equivalence is confirmed in all ten cases. EgoGazeLite achieves this at 15.7M parameters, 6.71 GFLOPs, and runs the full gaze-and-crop pipeline end-to-end in real time (21.6 ms/frame) on consumer accelerator hardware. Together, these results remove the need for eye-tracking hardware for token-efficient, gaze-conditioned egocentric video understanding with MLLMs.

多模态模型眼动预测轻量化移动计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。