InferLine
Daniel Crankshaw, Gur-Eyal Sela, Xiangxi Mo, Corey Zumar, Ion Stoica, Joseph E. Gonzalez, Alexey Tumanov
Abstract
Serving ML prediction pipelines spanning multiple models and hardware accelerators is a key challenge in production machine learning. Optimally configuring these pipelines to meet tight end-to-end latency goals is complicated by the interaction between model batch size, the choice of hardware accelerator, and variation in the query arrival process.
Topics & Concepts
Pipeline transportComputer scienceLatency (audio)Key (lock)Process (computing)Hardware accelerationOperating systemField-programmable gate arrayEngineeringTelecommunicationsEnvironmental engineeringMachine Learning and Data ClassificationData Stream Mining TechniquesMachine Learning and Algorithms