.png){kind=icon}
Return
Do Vision and Language Models Share Concepts? A Vector Space Alignment Study
.png){kind=icon}
OA
AI
DOI：10.1162/tacl_a_00698.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Large-scale pretrained language models (LMs) are said to lack the ability to connect utterances to the world (Bender and Koller, 2020), because they do not have mental models of the world (Mitchell and Krakauer, 2023). If so, one would expect LM representations to be unrelated to representations induced by vision models. We present an empirical evaluation across four families of LMs (BERT, GPT-2, OPT, and LLaMA-2) and three vision model architectures (ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge towards representations isomorphic to those of vision models, subject to dispersion, polysemy, and frequency. This has important implications for both multi-modal processing and the LM understanding debate (Mitchell and Krakauer, 2023).1
Journal
T
IF:
6.9
Papers:
486
Citations:
5.7K
Organization
Cited Papers
ShareSaveMatrix Effects in the Detection of Pb and Ba in Soils Using Laser-Induced Breakdown Spectroscopy
PREAI
ShareSaveShareSaveShareSaveInvestigating the lipid fingerprint of SLC6 neurotransmitter transporters: a comparison of dDAT, hDAT, hSERT, and GlyT2.png)
BBA Advances
OAAI
BBA AdvancesIF0
OAAIShareSaveShareSaveShareSaveShareSaveShareSave
View more