M<sup>3</sup>Track: Meta-Prompt for Multi-Modal Tracking
Zhangyong Tang, Tianyang Xu, Xiao‐Jun Wu, Josef Kittler
Abstract
Prompt-tuning has shown remarkable success in multi-modal visual tracking, which enhances RGB tracking by incorporating an additional modality, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i>, thermal infrared (T), Depth (D), or Event (E), forming an RGB+X tracking paradigm. However, in the testing phase, current approaches utilise a frozen prompt for the entire benchmark, failing to account for the diversity and unique characteristics of individual videos. To address this issue, we inject meta-learning solutions into current prompt-based tracking technique, thereby emphasising sequence-level adaptation. Unlike traditional prompt-based trackers, which keep the parameters of prompt blocks fixed during testing, our approach updates these parameters in the first frame of each video using a meta-learning solution. This allows for enhanced discriminative tracking capabilities tailored to each video. Building on this advancement, we extend our methodology beyond separate implementations for RGBT, RGBD, and RGBE tasks. A unified multi-modal tracker is further derived, resulting in the first unified tracker without any task priors (notification of task type) employed in both training and testing phases. Extensive experimental results on LasHeR, DepthTrack, VisEvent, GTOT, RGBT234, and RGBD1K consistently demonstrate the superiority of the proposed method against the existing prompt-tuning paradigm. Source codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Zhangyong-Tang/M3Track</uri>.