Hindi to English Multimodal Machine Translation on News Dataset in Low Resource Setting
Loitongbam Sanayai Meetei, Salam Michael Singh, Alok Singh, Ringki Das, Thoudam Doren Singh, Sivaji Bandyopadhyay
Abstract
This work proposes a multimodal Hindi-to-English machine translation on a news corpus by integrating multiple input modalities. The experimental dataset comprises of an image from a news article and its caption in the English-Hindi language. The parallel English-Hindi dataset is constructed by translating the image caption from English-language news articles into Hindi. Unlike standard images used for NLP tasks, images in news articles frequently depict significant events, locations, or individuals. The caption provides a more extensive description of the image, resulting in the inclusion of multiple identified entities. Our methodology consists of encoders for each input modality, allowing for simultaneous consideration of both text and image. We undertake two tests to examine how effectively the image content improves MT systems when 1) the entire image is considered first and 2) the largest object recognized in the image is considered. We employ Byte Pair Encoding (BPE) to represent the text, and the visual attributes are extracted from the image with VGG-19, a pre-trained CNN model. In terms of BLEU and chrF, our Multimodal Machine Translation (MMT) systems are superior than the baseline unimodal NMT system. Our MMT system outperforms the NMT model by +1.8 BLEU and +0.03 chrF.