SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing
Zeyinzi Jiang, Chaojie Mao, Yulin Pan, Zhen Han, Jingfeng Zhang
Abstract
Image diffusion models have been utilized in various tasks, such as text-to-image generation and controllable im-age synthesis. Recent research has introduced tuning meth-ods that make subtle adjustments to the original models, yielding promising results in specific adaptations of foun-dational generative diffusion models. Rather than modi-fying the main backbone of the diffusion model, we delve into the role of skip connection in U-Net and reveal that hi-erarchical features aggregating long-distance information across encoder and decoder make a significant impact on the content and quality of image generation. Based on the observation, we propose an efficient generative tuning framework, dubbed SCEdit, which integrates and edits Skip Connection using a lightweight tuning module named SC-Tuner. Furthermore, the proposed framework allows for straightforward extension to controllable image syn-thesis by injecting different conditions with Controllable SC-Tuner, simplifying and unifying the network design for multi-condition inputs. Our SCEdit substantially reduces training parameters, memory usage, and computational ex-pense due to its lightweight tuners, with backward propa-gation only passing to the decoder blocks. Extensive exper-iments conducted on text-to-image generation and control-lable image synthesis tasks demonstrate the superiority of our method in terms of efficiency and performance. Project page: https://scedit.github.io/