Irina: Accelerating DNN Inference with Efficient Online Scheduling
Xiaorui Wu, Hong Xu, Yi Wang
Abstract
DNN inference is becoming prevalent for many real-world applications. Current machine learning frameworks usually schedule inference tasks with the goal of optimizing throughput under predictable workloads and task arrival patterns. Yet, inference workloads are becoming more dynamic with bursty queries generated by various video analytics pipelines which run expensive inference only on a fraction of video frames. Thus it is imperative to optimize the completion time of these unpredictable queries and improve customer experience.
Topics & Concepts
Computer scienceInferenceScheduling (production processes)AnalyticsMachine learningTask (project management)ScheduleArtificial intelligenceData miningOperating systemOperations managementEconomicsManagementAdvanced Neural Network ApplicationsAnomaly Detection Techniques and ApplicationsData Stream Mining Techniques