Ad Lingua: Text Classification Improves Symbolism Prediction in Image Advertisements
A. Savchenko, Anton Alekseev, Sejeong Kwon, Elena Tutubalina, Evgeny Myasnikov, Sergey Nikolenko
Abstract
Understanding image advertisements is a challenging task, often requiring non-literal interpretation. We argue that standard image-based predictions are insufficient for symbolism prediction. Following the intuition that texts and images are complementary in advertising, we introduce a multimodal ensemble of a state of the art image-based classifier, a classifier based on an object detection architecture, and a fine-tuned language model applied to texts extracted from ads by OCR. The resulting system establishes a new state of the art in symbolism prediction.
Topics & Concepts
Computer scienceIntuitionClassifier (UML)Artificial intelligenceArchitectureNatural language processingContextual image classificationImage (mathematics)Pattern recognition (psychology)ArtPsychologyCognitive scienceVisual artsMultimodal Machine Learning ApplicationsTopic ModelingSentiment Analysis and Opinion Mining