Learning to Generate Urban Design Images From the Conditional Latent Diffusion Model
Xiaotang Cui, Xiao Feng, Siwen Sun
Abstract
With the rapid process of computer vision and deep learning, image synthesis models, such as the latent diffusion models, have exhibited remarkable performances in producing high-quality and realistic results. However, achieving precise layout control through adjusting text prompts solely proves to be challenging for the diffusion model. Therefore, we organize the conditional control network to instruct the latent diffusion model towards generating satisfactory results. Besides, direct training from scratch or fine-tuning the latent diffusion model on new datasets is non-trivial due to massive parameters. To tackle the troublesome training problem, we implement the low-rank adaptation strategy in the training process of the diffusion model. The low-rank adaptation strategy decomposes 2-dimensional matrices into 1-dimensional vectors, which can decrease the number of parameters greatly and accelerate the training of the latent diffusion model. To synthesize high-quality images, we collect urban design images from pinterest and generate homologous text prompts. We intend to make this dataset publicly available for further research and development in the field. Both qualitative and quantitative evaluations demonstrate the effectiveness and capacity of our framework.