arXiv:2607.00410cs.CVcs.LG2026-07

用脑电波精准控制人脸微表情,实现身份不变的精细编辑。

MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment

论文配图:MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment
图 1 · 摘自论文原文
  • 通过双流流形对齐,将脑电信号与面部动作单元对齐。
  • 在真实数据上实现90%以上的动作单元编辑准确率。
  • 适合脑机接口、情感计算及残障人士表达辅助研究者。

近期脑解码研究在从神经信号重构外部视觉内容方面取得显著进展,但利用脑电图(EEG)引导面部表情编辑仍处于探索阶段,面临独特挑战:不需恢复被试所见内容,而是从噪声干扰的EEG信号中识别与面部动作相关的模式,并将其锚定于局部化、保持身份一致的表情编辑中。本文提出MindAU,一种统一框架,用于从EEG信号控制细粒度面部动作单元(AU)编辑。MindAU首先通过时序掩码重建和AU分类监督学习鲁棒且具有区分性的EEG表示。随后,通过双流流形对齐,在Qwen2.5-VL多模态空间中将EEG特征与AU级文本语义及去身份化的视觉位移轨迹对齐。最后,引入脑电感知的多模态旋转位置编码、基于关键点的参考掩码和AU感知区域监督,构建基于扩散模型的高保真、身份保持的编辑器。我们还提出了E-CAFE,一个包含配对EEG-人脸编辑样本和标准化评估协议的精选基准。大量实验验证了MindAU的有效性,表明其在面向面部神经肌肉障碍患者的未来辅助表达技术中具有潜力。

原文摘要 · Abstract (English)

Recent brain decoding studies have made substantial progress in reconstructing externally perceived visual content from neural signals. However, using electroencephalography (EEG) recordings to guide facial expression editing remains largely unexplored and poses a distinct challenge: rather than recovering what a subject sees, it requires identifying facial-action related patterns from noisy EEG signals and grounding them in localized, identity-preserving expression edits. In this paper, we investigate EEG-conditioned facial image editing for fine-grained facial action unit (AU) control and propose MindAU, a unified framework for controlling facial AU edits from EEG signals. MindAU first learns noise-robust and AU-discriminative EEG representations through temporal masked reconstruction and AU classification supervision. It then bridges the modality gap via Dual-Stream Manifold Alignment, aligning EEG features with AU-level text semantics and identity-reduced visual displacement trajectories in the multimodal space of Qwen2.5-VL. Finally, MindAU incorporates EEG-aware Multimodal Rotary Positional Embeddings, landmark-guided reference masking, and AU-aware region supervision into a multimodal diffusion-based editor for high-fidelity identity-preserving editing. We also introduce E-CAFE, a curated benchmark for EEG-Conditioned Action-Unit Facial Editing with paired EEG-face editing samples and standardized evaluation protocols. Extensive experiments demonstrate the effectiveness of MindAU and suggest its potential as a step towards future assistive expression technologies for individuals with facial neuromuscular disorders.

脑机接口表情编辑多模态对齐扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。