.png){kind=icon}
.png){kind=icon}
Return
Prototype-based cross-modal object tracking
.png){kind=icon}
OA
AI
DOI：10.1016/j.inffus.2025.102941.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Cross-modal object tracking is an important research topic in the field of information fusion, and it aims to address imaging limitations in challenging scenarios by integrating switchable visible and near-infrared modalities. However, existing tracking methods face some difficulties in adapting to significant target appearance variations in the presence of modality switch. For instance, model update based tracking methods struggle to maintain stable tracking results during modality switching, leading to error accumulation and model drift. Template based tracking methods solely rely on the template information from first frame and/or last frame, which lacks sufficient representation ability and poses challenges in handling significant target appearance changes. To address this problem, we propose a prototype-based cross-modal object tracker called ProtoTrack, which introduces a novel prototype learning scheme to adapt to significant target appearance variations, for cross-modal object tracking. In particular, we design a multi-modal prototype to represent target information by multi-kind samples, including a fixed sample from the first frame and two representative samples from different modalities. Moreover, we develop a prototype generation algorithm based on two new modules to ensure the prototype representative indifferent challenges. The prototype evaluation module estimates the reliability of tracking results for each frame, determining whether to perform prototype extraction based on the tracking result. The prototype classification module predicts the modality state for each frame, facilitating the dynamic prototype updating of the associated modality samples. The multi-modal prototype forms a robust target representation under temporal variation and modality switch, and we integrate it into two tracking frameworks. Extensive experiments on the CMOTB dataset demonstrate the effectiveness and generalization of the proposed ProtoTrack against state-of-the-art methods.
Keywords:
Cross-modal object tracking
Prototype learning
Multi-modal prototype
Prototype evaluation
Prototype classification
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
15.5
Papers:
4.1K
Citations:
2.7W
Organization
No organization information available