Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine
Andrew Lamb, Yijie Shen, Daniël Heres, Jayjeet Chakraborty, Mehmet Ozan Kabak, Liang-Chi Hsieh, Chao Sun
Abstract
Apache Arrow DataFusion is a fast, embeddable, and extensible query engine written in Rust that uses Apache Arrow as its memory model. In this paper we describe the technologies on which it is built, and how it fits in long-term database implementation trends. We then enumerate its features, optimizations, architecture and extension APIs to illustrate the breadth of requirements of modern OLAP engines as well as the interfaces needed by systems built with them. Finally, we demonstrate open standards and extensible design do not preclude state-of-the-art performance using a series of experimental comparisons to DuckDB.
Topics & Concepts
Computer scienceArrowModular designInformation retrievalDatabaseOperating systemProgramming languageAdvanced Database Systems and QueriesData Management and AlgorithmsGraph Theory and Algorithms