ClimateSpark: An in-memory distributed computing framework for big climate data analytics

doi:10.1016/j.cageo.2018.03.011

ClimateSpark: An in-memory distributed computing framework for big climate data analytics

2018-06-01

34

PRE

AI

DOI：10.1016/j.cageo.2018.03.011

Original

Original request for help

Save

Abstract

En 中文

The unprecedented growth of climate data creates new opportunities for climate studies, and yet big climate data pose a grand challenge to climatologists to efficiently manage and analyze big data. The complexity of climate data content and analytical algorithms increases the difficulty of implementing algorithms on high performance computing systems. This paper proposes an in-memory, distributed computing framework, ClimateSpark, to facilitate complex big data analytics and time-consuming computational tasks. Chunking data structure improves parallel I/O efficiency, while a spatiotemporal index is built for the chunks to avoid unnecessary data reading and preprocessing. An integrated, multi-dimensional, array-based data model (ClimateRDD) and ETL operations are developed to address big climate data variety by integrating the processing components of the climate data lifecycle. ClimateSpark utilizes Spark SQL and Apache Zeppelin to develop a web portal to facilitate the interaction among climatologists, climate data, analytic operations and computing resources (e.g., using SQL query and Scala/Python notebook). Experimental results show that ClimateSpark conducts different spatiotemporal data queries/analytics with high efficiency and data locality. ClimateSpark is easily adaptable to other big multiple-dimensional, array-based datasets in various geoscience domains.

Keywords:

Big data

High performance computing

Array-based data model

Climate data analytics

Apache spark

Geospatial cyberinfrastructure

Cloud computing

ClimateSpark: An in-memory distributed computing framework for big climate data analytics

Abstract

AI Summary

Journal

Organization