Extreme Video Compression with Prediction Using Pre-trained Diffusion Models

Abstract

Diffusion models have achieved remarkable success in generating high quality image and video data. More recently, they have also been used for image compression with high perceptual quality. In this paper, we present a novel approach to extreme video compression leveraging the predictive power of diffusion-based generative models at the decoder. The conditional diffusion model at the decoder takes several compressed frames and generates the subsequent frames. Since the encoder can perfectly replicate the generation process, it can decide which frames to transmit and which one can be generated to achieve the desired quality level. Therefore, only image compression is applied at the encoder, while the time correlation among the frames is utilized at the decoder in the generation process, which transfer the complexity from the encoder to the decoder. Experimental results demonstrate the effectiveness of the proposed scheme compared to standard codecs such as H.264 and H.265 in the low bpp regime considering perceptual quality metrics such as the learned perceptual image patch similarity (LPIPS) and the Frechet video distance (FVD). While this framework is not yet competitive with state of the art video compression methods, the presented results show their potential, particularly in the very low bit rate regime, and we will highlight its various advantages that could make it a viable alternative in certain applications. Code is available at: https://github.com/ElesionKyrie/Extreme-Video-Compression-With-Prediction-Using-Pre-trainded-Diffusion-Models-

Topics & Concepts