Lessons learned from comparing C-CUDA and Python-Numba for GPU-Computing
Lena Oden
Abstract
Python as programming language is increasingly gaining importance, especially in data science, scientific, and parallel programming. It is faster and easier to learn than classical programming languages such as C. However, usability often comes at the cost of performance and applications written in Python are considered to be much slower than applications written in C or FORTRAN. Further, it does not allow the usage of GPUs-besides of pre-compiled libraries.However, the Numba package promises performance similar to C code for compute intensive parts of a Python application and it supports CUDA, which allows the use of GPUs inside a Python application.In this paper we compare the performance of Numba-CUDA and C -CUDA for different kinds of applications. For compute intensive benchmarks, the performance of the Numba version only reaches between 50% and 85% performance of the CCUDA version, despite the reduction operation, where the Numba version outperforms CUDA. Analyzing the PTX code and CUDA performance counters revealed that index-calculation is one limiting factor in Numba. Another problem is the type interference for single precision computations, as some values are computed in double precision. By optimizing this within the Numba package, the performance of Numba improves. However, C-CUDA applications still outperform the Numba versions. Further analysis with the CloverLeav Mini App shows that Numba performance further decreases for applications with multiple different compute kernels. The non-GPU part slows down these applications, due to the slow Python interpreter. This leads to a worse GPU utilization.Today Python is widely used in industry and academia and has been the first choice of coding languages among software programmers in the last years. Currently, according to the TIOBE index [5], it is the 3rd most popular programming language and the number one in IEEE Spectrum's fifth annual interactive ranking of the top programming languages [4]. One reason for this is that is easier to learn than classical programming languages like C. However, the other reason is the increasing popularity of Data Science, where Python is the most used language. A collection of libraries such as NumPy [22], and Matplotlib [1] or Scipy [8] provide a rich set of functions for scientific computing [16]. Packages like Dask [19], PyCompss [21] and MPI for Python [6] allow running Python applications on large, parallel machines, promising high performance. However, the performance of Python is considered slow compared to compiled languages such as C, C++, and FORTRAN, especially for heavy computations. In recent years, more and more tools have been developed to counter this prejudice. Numpy [22], for example, uses C-like arrays to store data and offers fast functions implemented in C to speed up calculations. The CuPy [14] package provides a similar set of functions, but these functions are implemented for GPUs using CUDA. The SciPy library is based on NumPy and provides a rich set on functionalities for scientific computing. Still, the high performance of these libraries is provided by the underling C-implementations. Internally, they use libraries like OpenBlas or IntelMKL to reach high performance and therefore, they are limited by the functions which are provided by theses libraries. Therefore, a performance problem always arises when the required functionality is not implemented within these libraries. In this case, the application falls back to the Python interpreter. Compared to "bare metal" code, interpreted code is slow. In addition, in Python it is not possible to use GPUs or other accelerators directly, as the Python interpreter cannot execute code on these machines. Therefore, the usage is only possible with precompiled libraries. To overcome this limitation, different approaches where developed to mix C, CUDA or OpenCL with Python. Cython [2] allows integrating C-code in Python applications to improve performance of critical sections. It also allows an easy development of wrappers for C-libraries. Similar, packages such as PyCuda and PyOpenCL [9] support wrappers for CUDA or OpenCL code within a Python script. Both approaches require the mixture of different programming languages.Numba [10] follows a different approach. Instead of merging C/CUDA code with Python, it allows the development of efficient applications for both, CPUs and GPUs in Python style. When a Python script using Numba is executed, marked functions are compiled just-in-time (JIT) using the LLVM framework. Using Python for GPU programming can mean a considerable simplification in the development of parallel applications.But often a simplification of comes at the expense of performance, and one expects a performance loss from Python compared to pure C code. In this paper, we want to understand the differences between native C-CUDA code and CUDA-code written in Python with Numba. We also want to share some basic tips how to improve the performance of applications written in Numba.We will first analyse a few micro benchmarks in detail. We are using these simple benchmarks, as it is easier to understand the differences with small code examples. We will use the collected information to derive some optimization for Numba. Finally, we evaluate and compare the performance of a more application like mini-app, written in C-CUDA and Numba accelerated Python. We will evaluate if our insights from the microbenchmarks to real applications.