Text to Image Generator with Latent Diffusion Models
Apoorva Rauniyar, Aryan Raj, Ashish Kumar, Ashish Kumar Kandu, Astha Singh, Anjani Gupta
Abstract
Generating images from natural language text descriptions is an impressive feat of deep learning. Diffusion models have enabled state-of-the-art synthesis results for various types of data, including images, by breaking down the image generation process into successive denoising autoencoder applications. However, these models can be computationally expensive and time-consuming to train and infer due to their reliance on pixel space. This paper introduces a novel approach to training diffusion models with limited computational resources while still maintaining quality and flexibility using pre-trained autoencoders. We have developed a methodology that strikes a near-perfect equilibrium between simplifying complexity and maintaining intricate details, enhancing visual accuracy. We also propose a cross-attention layer in our model design to enable high-resolution synthesis using a convolutionbased approach for conditioned inputs like text and bounding boxes. Our latent diffusion model exhibits exceptional performance in multiple image synthesis tasks such as text image synthesis, super-resolution, and unconditional picture generation while reducing computational requirements significantly compared to pixel-based models.