.png){kind=icon}
.png){kind=icon}
Return
Self-supervised cross-modal visual retrieval from brain activities
.png){kind=icon}
OA
AI
DOI：10.1016/j.patcog.2023.109915.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
We study the problem of restoring visual stimuli from visually-evoked electroencephalography (EEG) signals. Using a supervised classification-then-generation framework, the reconstruction-based approaches learn the mapping between distributions of two modalities but fail to reproduce the exact visual stimulus. Instead, we propose a self-supervised cross-modal retrieval paradigm that seeks instance-level alignment by maximizing the mutual information between the EEG encoding and associated visual stimulus. We demonstrate the threefold advantages of the self-supervised retrieval over supervised reconstruction on the largest visual-evoked EEG dataset with two evaluation protocols. First, it restores the exact visual stimulus without accessing the image class information, which was not possible with previous approaches. Second, it produces more recognizable results than generated ones and bypasses the challenge of training an image generator. Finally, it illustrates the benefits of self-supervision over supervised models in handling open-set data.
Keywords:
Visual stimuli recovery
Cross-modal retrieval
Self-supervised learning
Brain-Computer Interface
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.