An Empirical Study on the Energy Usage and Performance of Pandas and Polars Data Analysis Python Libraries
Felix Nahrstedt, Mehdi Karmouche, Karolina Bargieł, Pouyeh Banijamali, Apoorva Nalini Pradeep Kumar, Ivano Malavolta
Abstract
Context. Python’s growing popularity in data analysis and the contemporary emphasis on energy-efficient software tools necessitate an investigation into the energy implications of data operations, particularly in resource-intensive domains like data science. Goal. We aim to assess the energy usage of Pandas, a widely-used Python data manipulation library, and Polars, a Rust-based library known for its performance. The study aims to provide insights for data scientists by identifying scenarios where one library outperforms the other in terms of energy usage, while exploring the possible correlations between energy and performance metrics. Method. We performed four separate experiment blocks including 8 Data Analysis Tasks (DATs) from an official TPCH Benchmark done by Polars and 6 Synthetic DATs. Both DATs groups are run with small and large dataframes and for both libraries. Results. Polars is more energy-efficient than Pandas when manipulating large dataframes. For small dataframes, the TPCH Benchmarking DATs does not show significant differences, while for the Synthetic DATs, Polars performs significantly better. We identified strong positive correlations between energy usage and execution time, as well as memory usage for Pandas, while Polars did not show significant memory usage correlations for the majority of runs. There is a significantly negative correlation between energy usage and CPU usage for Pandas. Conclusions. We recommend using Polars for energy-efficient and fast data analysis, emphasizing the importance of CPU core utilization in library selection.