.png){kind=icon}
返回
Encoder-decoder cycle for visual question answering based on perception-action cycle
.png){kind=icon}
PRE
AI
DOI：10.1016/j.patcog.2023.109848.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
In this study, we propose a novel encoder-decoder cycle (EDC) framework inspired by the human learning process called the perception-action cycle to tackle challenging problems such as visual question answering (VQA) and visual relationship detection (VRD). EDC considers the understanding of the visual features of an image as perception and the act of answering the question regarding that image as an action. In the perception-action cycle, information is primarily collected from the environment and then passed to sensory structures in the brain to form an understanding of the environment. Acquired knowledge is then passed to motor structures to perform an action on the environment. Next, sensory structures perceive the altered environment and improve their understanding of the surrounding world. This process of understanding the environment, performing an action correspondingly, and then re-evaluating the initial understanding occurs cyclically in human life. EDC initially mimics this mechanism of introspection by comprehending and refining visual features to acquire the proper knowledge for answering the question. Subsequently, it decodes visual and language features into answer features, feeding them back cyclically to the encoder. In the VRD task, EDC decodes visual features to generate predicate features. We evaluate the proposed framework on the TDIUC, VQA 2.0, and VRD datasets, which outperforms the state-of-the-art models on the TDIUC and VRD datasets.
Keyword:
Visual question answering
Vision language tasks
Multi-modality fusion
Attention
Bilinear fusion
Brain-inspired frameworks
期刊
IF:
7.6
论文数:
1.3W
被引数:
4.5W
机构
引用论文
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations视觉基因组: 使用众包密集图像注释连接语言和视觉
OAAI
OAAI分享收藏分享收藏Visual Relationship Detection With Image Position and Feature Information Embedding and Fusion.png)
IEEE ACCESS
OAAI
IEEE ACCESSIF3.6
OAAI分享收藏Direct replication of Gervais & Norenzayan (2012): No evidence that analytic thinking decreases religious beliefPLOS ONEOAAI
PLOS ONEIF0
OAAI分享收藏Learning multimodal relationship interaction for visual relationship detectionPATTERN RECOGNITION
PATTERN RECOGNITIONIF7.6
PREAI
分享收藏Low-shot transfer with attention for highly imbalanced cursive character recognition低镜头转移注意高度不平衡的草书字符识别NEURAL NETWORKS
NEURAL NETWORKSIF6.3
PREAI
分享收藏
查看更多内容