A Brief Survey of Vector Databases

Abstract

The explosive growth of massive high-dimensional data requires capabilities for data processing, storing, and analyzing. This brings significant challenges to traditional databases due to the poor ability to handle high-dimensional data and its original design for stand-alone machines. Fortunately, vector databases have provided a practical solution for the management and analysis of high-dimensional data. Especially, they retrieve results related to the query efficiently after encoding various forms of data (e.g., text, image, and video) into vectors. The purpose of this paper is to offer insight into vector databases by presenting a brief survey. Firstly, the workflow of vector databases including indexing and querying, is detailed along with a specific case. Subsequently, we elaborate on the related methods applied in vector databases, which are the core techniques to enhance search efficiency and reduce computational overhead, particularly similarity search algorithms and similarity metrics. Further, we introduce widely used vector database products (e.g., Pinecone, Chroma, and Milvus) and compare them from multiple factors that should be taken into consideration. We also discuss potential avenues for future research in this domain. To conclude, this survey provides a comprehensive understanding of vector databases for retrieval from vast high-dimensional datasets.

Topics & Concepts