Image Caption Generator Using Attention Mechanism
Vaishnavi Agrawal, Shariva Dhekane, Neha Tuniya, Vibha Vyas
Abstract
Image captioning is used to generate sentences describing the scene captured in the form of images. It identifies objects in the image, performs a few operations, and tries to find the salient features of the image. Once the system identifies this information, it should further generate the most relevant and brief description for the image, which should be both syntactically and semantically correct. With the advancements of Learning techniques, algorithms can generate text in the form of natural sentences that will be able to describe an image in its best form. The natural ability of humans to understand image content and generate descriptive text is a challenging task for a machine to imitate. The applications of image caption generation are extensive and significant. The task involves generating brief captions using various techniques like Natural language processing (NLP), Computer vision (CV), and Deep Learning (DL) techniques. This paper introduces a system that uses an attention mechanism alongside an encoder and a decoder to generate the captions. It uses a pre-trained Convolutional Neural Network (CNN) viz. Inception V3 to extract features of the image and then a Recurrent Neural Network (RNN) viz. GRU to generate a relevant caption. To generate captions, the proposed model uses an attention mechanism that is Bahdanau attention. MS-COCO dataset is used to train the model. The results validate the model's reasonable ability to understand images and generate text.