INFless: a native serverless system for low-latency, high-throughput inference
Yanan Yang, Laiping Zhao, Yiming Li, Huanyu Zhang, Jie Li, Mingyang Zhao, Xingzhen Chen, Keqiu Li
Abstract
Modern websites increasingly rely on machine learning (ML) to improve their business efficiency. Developing and maintaining ML services incurs high costs for developers. Although serverless systems are a promising solution to reduce costs, we find that the current general purpose serverless systems cannot meet the low latency, high throughput demands of ML services.
Topics & Concepts
Computer scienceLatency (audio)ThroughputLow latency (capital markets)InferenceComputer networkTelecommunicationsArtificial intelligenceWirelessIoT and Edge/Fog ComputingAdvanced Data Storage TechnologiesCaching and Content Delivery