AugMMRev: An LLM-Augmented Multimodal Ranking Model for Personalized Image Material Retrieval
Yu Li, Jiaxuan He, Xiaoxiao Chen, Zhongyu Wang, Z. B. Chen, Niu Qi, Zulong Chen, Wenjian Xu, Yuyu Yin
Abstract
Consumer electronics devices—including smartphones and smart cameras—generate massive volumes of image data. Image retrieval serves as a critical enabling technology for diverse image-centric applications in consumer electronic applications (like AI-powered photo retrieval in smartphone, efficient media asset and creative template retrieval in smart camera, streaming recommendation systems for smart TVs). Nevertheless, text-query-based image retrieval encounters unique challenges within consumer electronics environments. First, in consumer electronics applications, text-to-image retrieving queries are typically concise, frequently leading to ambiguity in intent. Second,numerous images lack textual descriptions and metadata tags, while others contain inaccuracies or unreliable annotations—particularly in consumer-generated content where such semantic gaps critically undermine retrieval accuracy. Third, identical text queries elicit divergent expected results from distinct consumers across varying contexts. To address those challenges and improve image retrieval accuracy in consumer electronic applications, we propose AugMMRev. AugMMRev is primarily designed as a re-ranking model within a multimodal retrieval pipeline to evaluate the similarity between the query and a given candidate image. To achieve stronger re-ranking performance, we used a variety of processing methods. AugMMRev employs text augmentation to enrich abbreviated queries, utilizes image augmentation to enhance visually sparse content, and integrates user profiling to deliver personalized retrieval results. Comprehensive experiments are conducted to verify the effectiveness of AugMMRev. AugMMRev achieves 90.7% and 88.3% retrieval accuracy over real-business dataset and public dataset, which surpass the compared methods.