.png){kind=icon}
.png){kind=icon}
Return
A reference architecture for serverless big data processing
.png){kind=icon}
OA
AI
DOI：10.1016/j.future.2024.01.029.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Despite significant advances in data management systems in recent decades, the processing of big data at scale remains very challenging. While cloud computing has been well -accepted as a solution to address scalability needs, cloud configuration and operation complexity persist and often present themselves as entry barriers, especially for novice data analysts. Serverless computing and Function -as -a -Service (FaaS) platforms have been suggested to reduce such entry barriers by shifting configuration and operational responsibilities from the application developer to the FaaS platform provider. Naturally, serverless data processing (SDP)'', that is, using FaaS for (big) data processing, has received increasing interest in recent years. However, FaaS platforms were never intended to support large data processing tasks primarily. SDP, therefore, manifests itself through workarounds and adaptations on the application level, addressing various quirks and limitations of the FaaS platforms in use for data processing needs. This, in turn, creates tensions between the platforms and the applications using them, again encouraging the constant (re -)design of both. Consequently, we present lessons learned from a series of application and platform re -designs that address these tensions, leading to the development of an SDP reference architecture and a platform instantiation and implementation thereof called CREW. Mitigating the tensions through the process of application platform codesign proves to reduce both entry barriers and costs significantly. In some experiments, CREW outperforms traditional, non -SDP big data processing frameworks by factors.
Keywords:
Serverless data processing
Application platform co -design
Serverless reference architecture
Function as a Service
Software engineering
Cloud computing
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
F
IF:
6.1
Papers:
6.8K
Citations:
2.3W