.png){kind=icon}
.png){kind=icon}
Return
CrossProto-ViT: Unpaired Cross-Modal Prototype Alignment for Retinal Disease Diagnosis
.png){kind=icon}
PRE
AI
S
H
DOI：10.1007/s11760-026-05287-5.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
The accurate diagnosis of retinal pathologies is frequently impeded by the significant semantic gap and modality discrepancy between Color Fundus Photography (CFP) and Optical Coherence Tomography (OCT), particularly given the scarcity of paired clinical data. To address these challenges, CrossProto-ViT, a prototype-guided and semantically aligned multi-modal Vision Transformer framework, is presented. A hierarchical architecture is established by integrating Modality-Specific Stems and Retinal Modality Adapters within a shared ViT backbone to efficiently extract domain-specific features while preserving pre-trained knowledge. To bridge the domain gap in unpaired data, a cross-modal alignment mechanism is constructed utilizing a bidirectional Translator constrained by Conditional Optimal Transport and Cycle Consistency, coupled with an attention-based Cross-Fusion Module. Furthermore, a progressive three-stage training strategy encompassing prototype pretraining, alignment, and global optimization is implemented alongside an Adversarial Domain Discriminator to ensure robust convergence and modality invariance. Extensive experiments on heterogeneous multi-source datasets demonstrate that CrossProto-ViT achieves state-of-the-art performance, recording accuracies of 99.9% on OCT and 99.3% on CFP. The framework significantly outperforms leading baselines, including Swin Transformer V2 and the foundation model RETFound. Additionally, Grad-CAM visualizations substantiate superior lesion localization and semantic consistency, positioning CrossProto-ViT as a robust and interpretable solution for intelligent cross-modal retinal diagnosis.
Keywords:
Multi-modal deep learning
Vision Transformer
Retinal disease diagnosis
CFP
OCT
Journal
IF:
2.1
Papers:
778
Citations:
4.6K
Organization
Cited Papers