.png){kind=icon}
.png){kind=icon}
Return
Gradient-coupled cross-patch attention map for weakly supervised semantic segmentation
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2023.03.031.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Most advanced weakly supervised semantic segmentation studies have explored various strategies for refining the class activation mapping (CAM) maps and thus producing pixelwise pseudomasks for train-ing segmentation networks. However, CAM-based approaches suffer from the limited receptive fields of convolutional neural networks (CNNs) when exploring long-range feature dependencies, resulting in incomplete object region detection and inferior segmentation performance. On the other hand, the self-attention mechan ism has proven to be effective in modeling global contexts, providing a potential perspective for producing integral activation maps. Nevertheless, the use of self-attention maps to gen-erate pseudomasks faces significant obstacles, as the attention activations are class-agnostic, locally inconsistent and noisy. To address these issues, we propose a novel end-to-end transformer-based frame-work, namely the gradient-coupled cross-patch attention map (GC-PAM). First, the classification score gradients are backpropagated to recover the semantic knowledge of the attention map, which is then used to retrieve the object-relevant patches and object-irrelevant patches. Next, the corresponding object patch attention maps are aggregated through content-adaptive cross-patch coupling to construct com-prehensive activation maps, whereas the object-irrelevant maps are employed to suppress the back-ground noise. Subsequently, the resulting object activation maps are used to generate pseudolabels to supervise the segmentation branch. Extensive experiments are conducted to demonstrate the effective-ness of the proposed GC-PAM. Despite its simplicity and computational efficiency, the GC-PAM with the DeiT-S backbone surpasses the existing state-of-the-art approaches on the PASCAL VOC 2012 benchmark (75.3% val, 74.6% test). The results also demonstrate that the GC-PAM is a feasible alternative to CNN-based architectures.(c) 2023 Elsevier B.V. All rights reserved.
Keywords:
Weakly supervised semantic segmentation
Self-attention
Vision transformers
Class activation maps
Patch attention map