arrow
返回

Transparent serverless execution of Python multiprocessing applications

delete2023-03-01
delete6
delete
OA
AI
A
Aitor Arjona *
G
Gerard Finol
P
Pedro López
DOI:10.1016/j.future.2022.10.038delete
delete原文链接
delete分享
delete收藏
查看原文
摘要

摘要

En 中文
Access transparency means that both local and remote resources are accessed using identical op-erations. With transparency, unmodified single-machine applications could run over disaggregated compute, storage, and memory resources. Hiding the complexity of distributed systems through transparency would have great benefits, like scaling-out local-parallel scientific applications over flexible disaggregated resources in the Cloud.This paper presents a performance evaluation where we assess the feasibility of access transparency over state-of-the-art Cloud disaggregated resources for Python multiprocessing applications. We have interfaced the multiprocessing module with an implementation that transparently runs processes on serverless functions and uses an in-memory data store for shared state.To evaluate transparency, we run in the Cloud four unmodified applications: Uber Research's Evolution Strategies, Baselines-AI's Proximal Policy Optimization, Pandaral.lel's dataframe, and Scikit Learn's Hyperparameter tuning. We compare execution time and scalability of the same application running over disaggregated resources using our library, with the single-machine Python multipro-cessing libraries in a large VM. For equal resources, applications efficiently using message-passing abstractions achieve comparable results despite the significant overheads of remote communication. Other shared-memory intensive applications do not perform due to high remote memory latency.The results show that Python's multiprocessing library design is an enabler towards transparency: legacy applications using efficient disaggregated abstractions can transparently scale beyond VM limited resources for increased parallelism without changing the underlying code or architecture.(c) 2022 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
Keyword:
Transparency
Access transparency
Serverless
FaaS
Multiprocessing
Parallel programming
AI总结

AI总结

对已上传原文的论文进行重点信息的提取,主要内容包括:简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。

期刊

F
Future Generation Computer Systems-The International Journal of eScience
IF:
6.1
论文数:
6.9K
被引数:
2.8W

机构

U
Universitat Rovira i Virgili
学者数:
262
论文数: 118
被引数: 5.1K
引用论文

引用论文

暂无论文信息