Image Captioning With Controllable and Adaptive Length Levels
Ning Ding, Chaorui Deng, Mingkui Tan, Qing Du, Zhiwei Ge, Qi Wu
Abstract
Image captioning is one of the fundamental problems of computer vision and has drawn great attention over the years. However, most existing methods in image captioning focus on improving the quality of the image captions, while ignoring the ability of controlling the caption style. In this work, we aim to improve the controllability of image captioning methods, especially, by choosing to describe the image either roughly or in detail. We find this can be achieved by adding a simple length level embedding into existing models, which enables them to generate length-controllable captions describing the image at a specified level of detail, and further improve the diversity. On top of it, we propose to learn a length-level reranking transformer that captures the correlation between the semantic complexities of the image and text modalities, which can be used to select the most suitable length level for each image to make the captions informative while not being redundant. Moreover, when the length of the generated captions grows, existing methods usually suffer from a linearly increased computational complexity due to their autoregressive (AR) nature. To aid this, we devise a non-autoregressive (NAR) approach that generates captions in a length-irrelevant complexity. Besides, we propose a refinement-enhanced sequence training scheme and also adopt a sequence-level knowledge distillation technique for the training of our NAR model to bridge its performance gap with the AR models. In the experiments, our length-controllable models not only achieve SOTA performance in terms of caption quality on the MS COCO dataset but more importantly, generate controllable and diverse image captions. Specifically, our NAR model outperforms the AR baselines in terms of controllability and diversity, and also significantly improves the decoding efficiency for longer captions. By further applying the sequence-level training schemes, the caption quality of our NAR model improves clearly and is competitive with the state-of-the-art AR baselines.