A-DSP: An Adaptive Join Algorithm for Dynamic Data Stream on Cloud System

Junhua Fang

doi:10.1109/TKDE.2019.2947055

A-DSP: An Adaptive Join Algorithm for Dynamic Data Stream on Cloud System

2020-01-01

6

PRE

AI

DOI：10.1109/TKDE.2019.2947055

Original

Original request for help

Save

Abstract

En 中文

The join operations, including both equi and non-equi joins, are essential to the complex data analytics in the big data era. However, they are not inherently supported by existing DSPEs (Distributed Stream Processing Engines). The state-of-the-art join solutions on DSPEs rely on either complicated routing strategies or resource-inefficient processing structures, which are susceptible to dynamic workload, especially when the DSPEs face various join predicate operations and skewed data distribution. In this paper, we propose a new cost-effective stream join framework, named A-DSP (Adaptive Dimensional Space Processing), which enhances the adaptability of real-time join model and minimizes the resource used over the dynamic workloads. Our proposal includes: 1) a join model generation algorithm devised to adaptively switch between different join schemes so as to minimize the number of processing task required; 2) a load-balancing mechanism which maximizes the processing throughput; and 3) a lightweight algorithm designed for cutting down unnecessary migration cost. Extensive experiments are conducted to compare our proposal against state-of-the-art solutions on both benchmark and real-world workloads. The experimental results verify the effectiveness of our method, especially on reducing the operational cost under pay-as-you-go pricing scheme.

Keywords:

Task analysis

Routing

Data models

Heuristic algorithms

Adaptation models

Parallel processing

Computational modeling

Distributed stream join

theta-join

cost effective

A-DSP: An Adaptive Join Algorithm for Dynamic Data Stream on Cloud System

Abstract

AI Summary

Journal

Organization