AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learning
Ziqi Zhou, Shengshan Hu, Minghui Li, Hangtao Zhang, Yechao Zhang, Hai Jin
Abstract
Multimodal contrastive learning aims to train a general-purpose feature extractor, such as CLIP, on vast amounts of raw, unlabeled paired image-text data. This can greatly benefit various complex downstream tasks, including cross-modal image-text retrieval and image classification. Despite its promising prospect, the security issue of cross-modal pre-trained encoder has not been fully explored yet, especially when the pre-trained encoder is publicly available for commercial use.
Topics & Concepts
Computer scienceDownstream (manufacturing)Artificial intelligenceEncoderModalFeature (linguistics)Feature extractionAdversarial systemAutoencoderExtractorImage (mathematics)Feature learningNatural language processingMachine learningPattern recognition (psychology)Deep learningEngineeringLinguisticsProcess engineeringOperating systemPolymer chemistryPhilosophyChemistryOperations managementAdversarial Robustness in Machine LearningDomain Adaptation and Few-Shot LearningGeophysical Methods and Applications