A fast scalable distributed kriging algorithm using Spark framework

doi:10.1007/s41060-020-00215-3

A fast scalable distributed kriging algorithm using Spark framework

2020-04-03

8

PRE

AI

S

Sourangshu Bhattacharya

S

Soumya K. Ghosh

DOI：10.1007/s41060-020-00215-3

Original

Save

Abstract

En 中文

Environmental and climate models used for weather prediction require evenly spaced meteorological datasets at a very high spatial and temporal resolution to facilitate the analysis of recent climatic changes. However, due to the small number of weather stations available, often the data collected from them are scattered and inadequate for such model creation. For this reason, very high-resolution gridded meteorological surface is developed by interpolating the available scattered data points to fulfill the need of various ecological and climatic applications. Among various interpolation techniques, Ordinary Kriging (OK) is one of the most popular and widely used gridding methodologies with a sound statistical basis providing a possibility to obtain highly accurate results. However, OK interpolation on large unevenly spaced data points is computationally demanding and has a computational cost that scales as the cube of the number of data points as it involves multiplication and inversion of matrices of large cardinalities infeasible for computation on a single node. Additionally, its standard implementation involves complex model fitting and function minimization steps which make automatic kriging analysis from raw data a considerable challenge. Meanwhile, Apache Spark has emerged as a large-scale data processing engine with a dedicated Machine Learning Library (MLLib) for processing large matrices and thereby can be used for large-scale kriging analysis with considerable time. In this paper, we present a new fast distributed OK algorithm on Apache Spark framework and provide an efficient and simple distributed matrix inversion scheme to accelerate the execution of distributed OK algorithm. We have employed Strassen's direct method for matrix inversion and the acceleration is achieved by exploiting the symmetry nature of the variance-covariance matrix of the OK equation to invert the matrix. We show experimentally that our distributed inversion scheme enables us to invert a16,000x16,000 matrix with 51% and 38% less wall clock time than distributed Spark-basedLUand Strassen's inversion scheme, respectively.

Keywords:

Linear algebra

Matrix multiplication

Strassen's algorithm

Apache Spark

A fast scalable distributed kriging algorithm using Spark framework

Abstract

Journal

Organization

Cited Papers