.png){kind=icon}
返回
Implicit Ray Transformers for Multiview Remote Sensing Image Segmentation
.png){kind=icon}
OA
AI
DOI：10.1109/TGRS.2023.3285659.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The mainstream convolutional neural network (CNN)-based remote sensing (RS) image semantic segmentation approaches typically rely on massively labeled training data. Such a paradigm struggles with the problem of RS multiview scene segmentation with limited labeled views due to the lack of consideration of 3-D information within the scene. In this article, we propose implicit ray transformer (IRT) based on implicit neural representation (INR) for RS scene semantic segmentation with sparse labels (5% of the images being labeled). We explore a new way of introducing the multiview 3-D structure priors to the task for accurate and view-consistent semantic segmentation. The proposed method includes a two-stage learning process. In the first stage, we optimize a neural field to encode the color and 3-D structure of the RS scene based on multiview images. In the second stage, we design a ray transformer to leverage the relations between the neural field 3-D features and 2-D texture features for learning better semantic representations. Different from previous methods that only consider 3-D priors or 2-D features, we incorporate additional 2-D texture information and 3-D priors by broadcasting CNN features to different point features along the sampled ray. To verify the effectiveness of the proposed method, we construct a challenging dataset containing six synthetic sub-datasets collected from the Carla platform and three real sub-datasets from Google Maps. Experiments show that the proposed method outperforms the CNN-based methods and the state-of-the-art INR-based segmentation methods in quantitative and qualitative metrics. The ablation study shows that under a limited number of fully annotated images, the combination of the 3-D structure priors and 2-D texture can significantly improve the performance and effectively complete missing semantic information in novel views. Experiments also demonstrate that the proposed method could yield geometry-consistent segmentation results against illumination changes and viewpoint changes. Our data and code will be public.
Keyword:
Three-dimensional displays
Feature extraction
Semantics
Task analysis
Remote sensing
Transformers
Annotations
Implicit neural representation (INR)
remote sensing (RS)
semantic segmentation
transformer
期刊
IF:
8.6
论文数:
2.2W
被引数:
10.7W
机构
引用论文
Different Forms of Vigilance in Response to the Presence of Predators and Conspecifics in a Group‐Living Mammal, the European Rabbit.png)
Ethology
EthologyIF0
PREAI
分享收藏分享收藏Inflexibility of mental planning: A characteristic disorder with prefrontal lobe lesions?心理计划的僵化: 前额叶病变的特征性障碍？
PREAI
分享收藏Relation Matters: Relational Context-Aware Fully Convolutional Network for Semantic Segmentation of High-Resolution Aerial Images关系问题: 用于高分辨率航空图像语义分割的关系上下文感知全卷积网络
OAAI
OAAI分享收藏
查看更多内容