GSLICE
Aditya Dhakal, Sameer G. Kulkarni, K. K. Ramakrishnan
Abstract
The increasing demand for cloud-based inference services requires the use of Graphics Processing Unit (GPU). It is highly desirable to utilize GPU efficiently by multiplexing different inference tasks on the GPU. Batched processing, CUDA streams and Multi-process-service (MPS) help. However, we find that these are not adequate for achieving scalability by efficiently utilizing GPUs, and do not guarantee predictable performance.
Topics & Concepts
Computer scienceScalabilityCUDAGraphics processing unitCloud computingGeneral-purpose computing on graphics processing unitsGraphicsInferenceProcess (computing)MultiplexingComputer architectureParallel computingDistributed computingDatabaseComputer graphics (images)Operating systemArtificial intelligenceTelecommunicationsAdvanced Neural Network ApplicationsGraph Theory and AlgorithmsAdvanced Image and Video Retrieval Techniques