.png){kind=icon}
.png){kind=icon}
Return
TokenPacker: Efficient Visual Projector for Multimodal LLM
.png){kind=icon}
PRE
AI
DOI：10.1007/s11263-025-02491-7.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
In multimodal large language models (MLLMs), the visual projector is a crucial component that connects the visual encoder with the large language model (LLM). Most current MLLMs adopt a simple multi-layer perceptron (MLP) to preserve visual contexts via direct transformation. However, this approach tends to generate redundant visual tokens, particularly when processing high-resolution images, ultimately reducing the efficiency of MLLMs. Recent efforts to address this issue have employed resamplers or abstractors to reduce token quantity. Unfortunately, these methods often fail to capture finer details, thereby limiting the model’s visual reasoning capabilities. In this work, we introduce TokenPacker, a novel visual projector designed to generate condensed visual tokens through a coarse-to-fine scheme. Initially, we interpolate the visual features into a low-resolution point query that provides an overall visual representation. We then integrate high-resolution, multi-level regional cues using a region-to-point injection module, which enriches the point query with local context. This enhancement effectively transforms the initial query into a more detailed representation suitable for LLM reasoning. Furthermore, we propose a dynamic image slicing scheme to efficiently handle high-resolution images with TokenPacker. Extensive experiments demonstrate that TokenPacker can compress the visual tokens by 75% $$\sim $$ 89%, while maintaining or even improving performance on various benchmarks, achieving significantly higher efficiency. The source codes and models can be found at https://github.com/CircleRadon/TokenPacker .
Keywords:
Visual projector
Token compression
High-resolution MLLM
Journal
IF:
9.3
Papers:
3.9K
Citations:
2.8W