Consensus Knowledge Exploitation for Partial Query Based Image Retrieval
Yan Zhang, Zhong Ji, Yanwei Pang, Xuelong Li
Abstract
Partial Query based Image Retrieval (PQIR) enables a search engine to perform an interactive retrieval given by only an initial query and actively provide alternative feedbacks for a user to refine a set of retrieval results. It alleviates the deficiency in practice interactive image retrieval that requires the user to laboriously provide detailed feedbacks, and enables the retrieval on-the-fly with the incomplete initial query. Although significant progress has been made, existing works remain have challenge in actively providing more discriminative feedbacks. To address this challenge, we propose a novel Attributes&Objects-based Consensus Extraction and Representation (AoCer) framework. Specifically, we formulate a simple but effective Attribute&Object Feedback (AOF) paradigm, which employs both attributes and objects as intermediate feedbacks to carry out multiple rounds of interaction. To mine the intrinsic associations among concepts and enhance their feature representations, we further propose an Interventional Consensus Representation Learning (ICRL) module, which mainly constructs an interventional concept graph to yield the Interventional Consensus Representation (ICR). In addition, a Dual-Head Feedback Sampler (DHFS) is developed to sample objects and attributes for conducting the next round retrieval. Extensive experiments demonstrate the superiority of the proposed framework. Our source code will be released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zhangy0822/AoCer</uri> .