Super Cloud Library

Super Cloud Library

Welcome to the Mesoscale Dynamics & Modeling Group's Super Cloud Library. This is a NASA AIST funded project for developing tools to subset, analyze, visualize and distribute high-resolution, long-term cloud-resolving model (CRM) data using NCCS Hadoop clusters using a web interface. For the current project, we have developed an approach using the Comma-Separated Value (CSV) data format, which is used to convert NetCDF data, to store and process data on the Hadoop Distributed File System (HDFS). We have optimized this approach and reduced data sizes by using a common indexing file for multiple variable files. This CSV-based approach has enabled us to adaptively subset and visualize data with the tools available in the Hadoop ecosystem such as HIVE, Impala, Hue, and Spark. The technology has been applied to NASA-Unified Weather Research and Forecasting (NU-WRF) and Goddard Cumulus Ensemble (GCE) model data.