Composing high-level stream processing pipelines

Tanmaya Mahapatra

doi:10.1186/s40537-020-00353-2

Return

Composing high-level stream processing pipelines

2020-09-29

Tanmaya Mahapatra *

DOI：10.1186/s40537-020-00353-2

Original

Save

View PDF

Abstract

En 中文

The growing number of Internet of Things (IoT) devices provide a massive pool of sensing data. However, turning data into actionable insights is not a trivial task, especially in the context of IoT, where application development itself is complex. The process entails working with heterogeneous devices via various communication protocols to co-ordinate and fetch datasets, followed by a series of data transformations. Graphical mashup tools, based on the principles of flow-based programming paradigm, operating at a higher-level of abstraction are in widespread use to support rapid prototyping of IoT applications. Nevertheless, the current state-of-the-art mashup tools suffer from several architectural limitations which prevent composing in-flow data analytics pipelines. In response to this, the paper contributes by (i) designing novel flow-based programming concepts based on the actor model to support data analytics pipelines in mashup tools, prototyping the ideas in a new mashup tool called aFlux and providing a detailed comparison with the existing state-of-the-art and (ii) enabling easy prototyping of streaming applications in mashup tools by abstracting the behavioural configurations of stream processing via graphical flows and validating the ease as well as the effectiveness of composing stream processing pipelines from an end-user perspective in a traffic simulation scenario.

Keywords:

Flow-based programming

Graphical pipelines

Mashup tools

Graphical stream processing

Stream analytics

End-user programming

Data Analytics as a Service (DAaaS)

Data Analytics Applications (DAAs)

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.