.png){kind=icon}
返回
dispel4py: A Python framework for data-intensive scientific computing
.png){kind=icon}
OA
AI
DOI：10.1177/1094342016649766.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
This paper presents dispel4py, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. These combine the familiarity of Python programming with the scalability of workflows. Data streaming is used to gain performance, rapid prototyping and applicability to live observations. dispel4py enables scientists to focus on their scientific goals, avoiding distracting details and retaining flexibility over the computing infrastructure they use. The implementation, therefore, has to map dispel4py abstract workflows optimally onto target platforms chosen dynamically. We present four dispel4py mappings: Apache Storm, message-passing interface (MPI), multi-threading and sequential, showing two major benefits: a) smooth transitions from local development on a laptop to scalable execution for production work, and b) scalable enactment on significantly different distributed computing infrastructures. Three application domains are reported and measurements on multiple infrastructures show the optimisations achieved; they have provided demanding real applications and helped us develop effective training. The dispel4py.org is an open-source project to which we invite participation. The effective mapping of dispel4py onto multiple target infrastructures demonstrates exploitation of data-intensive and high-performance computing (HPC) architectures and consistent scalability.
Keyword:
Data-intensive computing
e-infrastructures
data streaming
scientific workflows
programming frameworks
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
2.5
论文数:
1.1K
被引数:
1.3K
机构
引用论文
分享收藏Implementation of a Reuse Process for Liquid Crystal Displays Using an Eccentric-Form Tool使用偏心形式工具实现液晶显示器的重用过程OAAI
OAAI分享收藏Streamline: scheduling streaming applications in a wide area environment.png)
MULTIMEDIA SYSTEMS
MULTIMEDIA SYSTEMSIF3.1
PREAI
分享收藏Workflows for automated downstream data analysis and visualization in large-scale computational mass spectrometry大规模计算质谱中自动化下游数据分析和可视化的工作流程PROTEOMICSOAAI
PROTEOMICSIF3.9
OAAI分享收藏
查看更多内容