Accelerated Neural Enhancement for Video Analytics With Video Quality Adaptation

Abstract

The quality of the video stream is the key to neural network-based video analytics. However, low-quality video is inevitably collected by existing surveillance systems because of poor-quality cameras or over-compressed/pruned video streaming protocols, e.g., as a result of upstream bandwidth limit. To address this issue, existing studies use quality enhancers (e.g., neural super-resolution) to improve the quality of videos (e.g., resolution) and eventually ensure inference accuracy. Nevertheless, directly applying quality enhancers does not work in practice because it will introduce unacceptable latency. In this paper, we present AccDecoder, a novel accelerated decoder for real-time and neural-enhanced video analytics, selects a few frames adaptively via Deep Reinforcement Learning (DRL) to enhance the quality and inference then reuse on the unselected ones. Next, we extend AccDecoder to AccDecoder <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$+$</tex-math> </inline-formula> by formulating the resolution-involved Markov decision process (MDP) to achieve resolution adaptation; it aims to trade accuracy and latency corresponding under various video resolutions. Proved by experiments, AccDecoder provides efficient inference capability via filtering important frames using DRL for DNN-based inference and reusing the results for the other frames via extracting the reference relationship among frames and blocks, which contributes 6-21% accuracy improvement and a latency reduction of 20-80% than baselines. Compared with AccDecoder, AccDecoder <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$+$</tex-math> </inline-formula> achieves an additional 2-7% accuracy improvement.

Topics & Concepts