.png){kind=icon}
Return
Encoder-decoder cycle for visual question answering based on perception-action cycle
.png){kind=icon}
PRE
AI
DOI：10.1016/j.patcog.2023.109848.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
In this study, we propose a novel encoder-decoder cycle (EDC) framework inspired by the human learning process called the perception-action cycle to tackle challenging problems such as visual question answering (VQA) and visual relationship detection (VRD). EDC considers the understanding of the visual features of an image as perception and the act of answering the question regarding that image as an action. In the perception-action cycle, information is primarily collected from the environment and then passed to sensory structures in the brain to form an understanding of the environment. Acquired knowledge is then passed to motor structures to perform an action on the environment. Next, sensory structures perceive the altered environment and improve their understanding of the surrounding world. This process of understanding the environment, performing an action correspondingly, and then re-evaluating the initial understanding occurs cyclically in human life. EDC initially mimics this mechanism of introspection by comprehending and refining visual features to acquire the proper knowledge for answering the question. Subsequently, it decodes visual and language features into answer features, feeding them back cyclically to the encoder. In the VRD task, EDC decodes visual features to generate predicate features. We evaluate the proposed framework on the TDIUC, VQA 2.0, and VRD datasets, which outperforms the state-of-the-art models on the TDIUC and VRD datasets.
Keywords:
Visual question answering
Vision language tasks
Multi-modality fusion
Attention
Bilinear fusion
Brain-inspired frameworks
Journal
IF:
7.6
Papers:
1.3W
Citations:
4.5W
Organization
Cited Papers
ShareSaveVisual Relationship Detection With Image Position and Feature Information Embedding and Fusion.png)
IEEE ACCESS
OAAI
IEEE ACCESSIF3.6
OAAIShareSaveDirect replication of Gervais & Norenzayan (2012): No evidence that analytic thinking decreases religious beliefPLOS ONE
OAAI
PLOS ONEIF0
OAAIShareSaveShareSaveLearning multimodal relationship interaction for visual relationship detectionPATTERN RECOGNITION
PATTERN RECOGNITIONIF7.6
PREAI
ShareSaveLow-shot transfer with attention for highly imbalanced cursive character recognitionNEURAL NETWORKS
NEURAL NETWORKSIF6.3
PREAI
ShareSave
View more