Part-Based Semantic Transform for Few-Shot Semantic Segmentation

doi:10.1109/TNNLS.2021.3084252

Part-Based Semantic Transform for Few-Shot Semantic Segmentation

2022-12-01

25

PRE

AI

DOI：10.1109/TNNLS.2021.3084252

Original

Original request for help

Save

Abstract

En 中文

Few-shot semantic segmentation remains an open problem for the lack of an effective method to handle the semantic misalignment between objects. In this article, we propose part-based semantic transform (PST) and target at aligning object semantics in support images with those in query images by semantic decomposition-and-match. The semantic decomposition process is implemented with prototype mixture models (PMMs), which use an expectation-maximization (EM) algorithm to decompose object semantics into multiple prototypes corresponding to object parts. The semantic match between prototypes is performed with a min-cost flow module, which encourages correct correspondence while depressing mismatches between object parts. With semantic decomposition-and-match, PST enforces the network's tolerance to objects' appearance and/or pose variation and facilities channelwise and spatial semantic activation of objects in query images. Extensive experiments on Pascal VOC and MS-COCO datasets show that PST significantly improves upon state-of-the-arts. In particular, on MS-COCO, it improves the performance of five-shot semantic segmentation by up to 7.79% with a moderate cost of inference speed and model size. Code for PST is released at https://github.com/Yang-Bob/PST.

Keywords:

Semantics

Image segmentation

Prototypes

Training

Measurement

Transforms

Feature extraction

Few-shot segmentation

prototype mixture models (PMMs)

semantic match

semantic transform

Part-Based Semantic Transform for Few-Shot Semantic Segmentation

Abstract

AI Summary

Journal

Organization