Design and implementation of a BigQuery dataset and application programmer interface (API) for the U.S. National Water Model
Kel Markert, Guilherme Gonçalves da Silva, Daniel P. Ames, Iman Maghami, Gustavious P. Williams, E. James Nelson, James Halgren, Arpita Patel, Adler Santos, Michael J. Ames
Abstract
We introduce an open-source web-based Application Programming Interface (API) developed within a representational state transfer (REST) architecture framework that provides access to the operational streamflow forecasts from the U.S. National Water Model (NWM). We built this API within the Google Cloud infrastructure, taking advantage of Google's API Gateway, BigQuery, and the Google Cloud Run architecture. We ran a data transformation from netCDF to tabular formation then ingested data into BigQuery using Apache Beam parallel processing technology. The API activates functions deployed to Cloud Run that executes SQL queries within the BigQuery data warehouse. The API gives users granular control to specify queries based on forecast type, reference datetime, stream segment specifics, and forecast ensemble members. This API greatly simplifies access and use of current and historic NWM forecasts, an otherwise arduous task due to the large volume of data and unwieldy storage methods. Retrieved forecast data can be used for many critical water management applications providing actionable intelligence for, e.g., flood management, safety, recreation, agriculture, and related uses.